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From  the  Chairman 
and  Executive  Editor 


The  theme  for  this  edition  of  the  Defense  Acquisition 
Research  Journal  is  “The  Art  of  Managing  Choices  ”  as  the 
articles  contained  herein  provide  tools,  techniques,  and 
even  food  for  thought  when  it  comes  to  decision  making. 


The  first  article,  “Valuation  of  Real  Options  as  Competitive 
Prototyping  in  System  Development,”  by  Diana  1.  Angelis;  David  N.  Ford; 
and  COL  John  T.  Dillard,  USA  (Ret.),  explains  how  the  use  of  a  finan¬ 
cial  tool  called  the  real  option  valuation  model  can  be  used  to  evaluate 
choices  for  major  weapons  systems,  despite  the  lack  of  monetary-based 
benefits.  In  the  second  article,  “Better  Buying  Power  or  Better  Off  Not? 
Purchasing  Technical  Data  for  Weapon  Systems,”  James  Hasik  proposes 
an  economic  model  that  will  allow  program  managers  to  price  technical 
data  packages,  to  help  choose  whether  and  how  to  purchase  them. 


“Initial  Capabilities  Documents:  A  10-Year  Retrospective  of  Tools, 
Methodologies,  and  Best  Practices”  by  Maj  Bryan  D.  Main,  USAF;  Capt 
Michael  P.  Kretser,  USAF;  Joshua  M.  Shearer;  and  Lt  Col  Darin  A.  Ladd, 
USAF,  distills  the  methodologies,  format,  and  necessary  content  that 
decision  makers  and  implementers  favor  to  create  more  effective,  con¬ 
cise,  and  complete  Initial  Capabilities  Documents.  Rounding  out  this 
edition,  in  “A  Proposed  2025  Ground  Systems  'Systems  Engineering' 
Process,”  the  authors  Robert  E.  Smith  and  LTC  Brian  D.  Vogt,  USA,  argue 
that  virtual  environments  and  new  manufacturing  methods  will  be  key 
to  decision  making  in  the  21st  century,  allowing  soldiers  and  engineers 
to  codevelop  tactics  and  tailored  ground  systems. 


The  featured  book  in  this  issue's  Defense  Acquisition  Professional 
Reading  List  is  the  provocatively  titled  Grounded:  The  Case  for  Abolishing 
the  United  States  Air  Force  by  Robert  M.  Farley,  which  argues  for  a 
game-changing  choice  regarding  the  DoD's  newest  Service.  Aleisha 
R.  Jenkins-Bey,  assistant  editor  for  the  Defense  Acquisition  Research 
Journal,  reviewed  the  book. 

Finally,  I  encourage  prospective  authors  to  consider  submitting  their 
manuscripts  for  the  Defense  Acquisition  University  Alumni  Association's 
2015  Acquisition  Symposium,  following  the  guidelines  in  the  Call  for 
Papers  in  this  issue.  The  deadline  for  submission  is  December  5,  2014. 


Dr.  Larrie  D.  Ferreiro 
Executive  Editor 
Defense  ARJ 


DAU  Center  for 
Defense  Acquisition 
Research 

Research  Agenda  2014 


The  Defense  Acquisition  Research  Agenda  is  intended  to  make 
researchers  aware  of  the  topics  that  are,  or  should  be,  of  partic¬ 
ular  concern  to  the  broader  defense  acquisition  community 
throughout  the  government,  academic,  and  industrial  sectors. 
The  purpose  of  conducting  research  in  these  areas  is  to  provide 
solid,  empirically  based  findings  to  create  abroad  body  of  knowl¬ 
edge  that  can  inform  the  development  of  policies,  procedures,  and 
processes  in  defense  acquisition,  and  to  help  shape  the  thought 
leadership  for  the  acquisition  community. 

Each  issue  of  the  Defense  ARJ  will  include  a  different  selection 
of  research  topics  from  the  overall  agenda,  which  is  at:  http:// 
www.dau.mil/research/Pages/researchareas.aspx 

Affordability  and  cost  growth 

•  Define  or  bound  “affordability”  in  the  defense  portfolio.  What  is  it? 
How  will  we  know  if  something  is  affordable  or  unaffordable? 

•  What  means  are  there  (or  can  be  developed)  to  measure,  manage,  and 
control  “affordability”  at  the  program  office  level?  At  the  industry 
level?  How  do  we  determine  their  effectiveness? 

•  What  means  are  there  (or  can  be  developed)  to  measure,  manage, 
and  control  “Should  Cost”  estimates  at  the  Service,  Component, 
program  executive,  program  office,  and  industry  levels?  How  do  we 
determine  their  effectiveness? 

•  What  means  are  there  (or  can  be  developed)  to  evaluate  and  compare 
incentives  for  achieving  “Should  Cost”  at  the  Service,  Component, 
program  executive,  program  office,  and  industry  levels? 


Recent  acquisition  studies  have  noted  the  vast  number  of  programs 
and  projects  that  do  not  make  it  successfully  through  the  acquisition 
system  and  are  subsequently  cancelled.  What  would  systematic 
root  cause  analyses  reveal  about  the  underlying  reasons,  whether 
and  how  these  cancellations  are  detrimental,  and  what  acquisition 
leaders  might  do  to  rectify  problems? 

Do  Joint  programs— at  the  inter-Service  and  international  levels— 
result  in  cost  growth  or  cost  savings  compared  with  single-Service 
(or  single -nation)  acquisition?  What  are  the  specific  mechanisms 
for  cost  savings  or  growth  at  each  stage  of  acquisition?  Do  the  data 
support  “jointness”  across  the  board,  or  only  at  specific  stages  of  a 
program,  e.g.,  only  at  research  and  development  or  only  with  specific 
aspects,  e.g.,  critical  systems  or  logistics? 

Can  we  compare  systems  with  significantly  increased  capability 
developed  in  the  commercial  market  to  DoD-developed  systems  of 
similar  characteristics? 

Is  there  a  misalignment  between  industry  and  the  government 
priorities  that  causes  the  cost  of  such  systems  to  grow  significantly 
faster  than  infiation? 

If  so,  can  we  identify  why  this  misalignment  arises?  What  relation¬ 
ship  (if  any)  does  it  have  to  industry's  required  focus  on  shareholder 
value  and/or  profit,  versus  the  government's  charter  to  deliver 
specific  capabilities  for  the  least  total  ownership  costs? 
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A  Real  Options  Valuation  Model  is  developed  to  recom¬ 
mend  how  to  valuate  technology  when  benefits  cannot 
be  measured  in  monetary  value.  Expected  values  of 
effectiveness  are  used  to  select  the  preferred  alterna¬ 
tive.  The  methodology  is  illustrated  using  three  guidance 
system  technologies  in  the  Army’s  Javelin  program.  The 
strategy  created  multiple  real  options  that  gave  the  Army 
the  right  (without  the  obligation)  to  select  one  guidance 
system  technology  based  on  the  outcome  of  technology 
development  tests.  Results  indicate  the  Army  paid 
less  than  the  total  value  of  the  options,  but  could  have 
increased  net  savings  by  paying  different  amounts  to  test 
each  alternative.  The  analysis  method  provides  a  logical 
and  defendable  approach  to  the  analysis  of  alternatives 
under  technology  development  uncertainty. 
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Real  options  can  be  described  along  several 
dimensions^  including  ownership^  the  source  of 
valuCy  complexity^  and  the  degree  to  which  the 
option  is  available. 


Competitive  prototyping  along  the  path  of  technology  development 
has  long  been  an  important  aspect  of  acquisition  program  strategies. 
Emphasis  in  this  method  reemerged  in  policy  documents  again  in  2007 
as  a  way  to  reduce  technical,  cost,  and  schedule  risk  by  leveraging  the 
economic  forces  of  competition  and  innovation  diversity  (Young,  2007). 
Academics  understand  these  fundamentals  and  their  multiple  benefits 
as  “real  options.”  This  article  describes  the  application  of  a  Real  Options 
Valuation  Model  to  the  three  candidate  guidance  technologies  during 
the  early  development  of  the  Javelin  anti-tank  missile  system.  A  short 
introduction  to  Real  Options  Theory  is  followed  by  a  description  of  the 
Javelin  guidance  technology  options.  Next,  a  model  for  measuring  the 
effectiveness  of  the  three  guidance  technologies  is  presented  and  the  cost 
effectiveness  of  each  alternative  based  on  “cost  per  kill”  under  determin¬ 
istic  and  probabilistic  assumptions  examined.  Finally,  a  decision  tree 
approach  is  used  to  model  the  value  of  each  option,  given  the  probability 
of  success  and  the  costs  to  recover  from  failure. 

Real  Options 

Real  Options  Theory  is  one  means  of  structuring  and  valuing  flexible 
strategies  to  address  uncertainty  (Courtney,  Kirkland,  &  Viguerie,  1997). 
An  option  is  a  right  without  an  obligation  to  take  specific  future  actions 
depending  on  how  uncertain  conditions  evolve  (Brealey  &  Meyers,  2000). 
Real  options  apply  Real  Options  Theory  to  tangible  assets.  The  central 
premise  of  Real  Options  Theory  is  that,  if  future  conditions  are  uncertain 
and  changing  the  strategy  later  incurs  substantial  costs,  then  having 
flexible  strategies  and  delaying  decisions  can  add  value  when  compared 
to  making  all  strategic  decisions  during  preproject  planning  (Amram  & 
Kulatilaka,  1999a,  1999b).  By  providing  managers  with  tools  to  respond 
to  changing  conditions,  real  options  can  increase  benefits,  limit  costs,  or 
both.  When  used  to  limit  costs  of  development  programs  such  as  in  the 
current  work,  real  options  are  a  form  of  risk  management. 
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The  design  of  an  option  compares  one  or  more  alternative  strategies 
that  may  be  used  in  the  future  to  a  reference  strategy  that  is  committed 
to  in  the  present.  To  use  an  option,  conditions  are  monitored  and  poten¬ 
tially  converted  into  a  signal  that  is  compared  to  trigger  conditions  using 
an  Exercise  Decision  Rule.  The  result  is  a  recommendation  on  whether 
to  abandon  the  reference  strategy  and  adopt  the  alternative  strategy 
(to  “exercise”  the  option)  or  to  keep  using  the  reference  strategy.  For 
example,  in  stock  purchase  options,  the  Exercise  Decision  Rule  is  to  buy 
a  stock  if  the  price  rises  above  a  certain  price  and  the  exercise  signal  is 
the  stock  price.  The  decision  delay  is  incurred  while  the  option  holder 
waits  to  see  if  the  stock  price  rises  above  the  exercise  price.  Options 
help  managers  learn  by  having  them  wait  to  see  (through  monitoring) 
how  uncertainty  evolves,  thereby  helping  them  make  better  choices. 
Therefore,  delayed  decision  making  is  an  important  feature  of  real 
options.  To  use  real  options,  one  must  define  the  exercise  signal  and 
exercise  decision  in  the  context  of  a  set  of  observable  variables  and  the 
exercise  of  the  option  in  operational  terms. 

Real  options  can  be  described  along  several  dimensions,  including 
ownership,  the  source  of  value,  complexity,  and  the  degree  to  which  the 
option  is  available.  A  common  topology  categorizes  real  options  accord¬ 
ing  to  the  type  of  managerial  action  applied,  including  options  that 
postpone  (holding  and  phasing  options),  change  the  amount  of  invest¬ 
ment  (growth,  scaling,  and  abandonment  options),  or  alter  the  form  of 
involvement  (switching  options).  The  study  of  real  options  can  focus  on 
the  monetary  valuation  of  the  flexibility  or  on  the  design  and  impacts  of 
real  options  on  decision  making  in  practice  (managerial  real  options). 
Both  of  these  aspects  of  real  options  can  improve  program  manage¬ 
ment  and  add  value.  Although  some  real  options  can  be  purchased  and 
exercised  at  low  cost  (e.g.,  the  option  to  have  salaried  employees  work 
overtime),  decisions  about  real  options  become  challenging  when  sig- 
niflcant  costs  are  required  to  obtain,  maintain,  or  exercise  the  flexibility 
that  may  add  value.  The  option  cost  is  what  must  be  paid  to  acquire  the 
flexibility  to  change  the  strategy.  Option  maintenance  costs  include 
benefits  lost  by  delaying  the  decision  and  costs  to  keep  the  flexibility 
available.  Option  exercise  costs  are  the  costs  of  changing  the  strategy. 
A  wide  variety  of  mathematical  models  have  been  developed  to  estimate 
the  monetary  value  of  options.  These  models  use  the  characteristics  of 
the  asset  and  the  benefits  and  costs  of  an  option  to  estimate  its  value. 
Option  valuation  began  with  efforts  to  price  options  on  financial  assets 
(e.g.,  shares  of  stock,  bonds)  and  other  market-traded  assets.  Initially, 
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closed-form  solutions  to  very  specific  situations  with  stringent  limits 
were  developed  (e.g.,  Black  &  Scholes,  1973).  Later  models,  such  as  the 
binomial  lattice  model  (Cox,  Ross,  &  Rubinstein,  1979),  were  developed 
that  could  be  used  to  value  a  wider  variety  of  options  and  assets. 


Real  Options  Valuation  Models  have  been 
effectively  used  to  demonstrate  how  real  options 
can  increase  project  value,  including  through 
engineering  design.... 


The  valuation  of  real  options  differs  from  that  of  financial  options  in 
that  the  underlying  assets  are  real  assets  that  are  often  not  traded  and 
represent,  for  example,  contingent  decisions  to  delay,  abandon,  expand, 
contract,  or  switch  project  components  or  methods.  Conventionally, 
researchers  have  estimated  the  value  of  real  options  based  on  approaches 
used  to  value  financial  options.  Real  options  valuation  methods  have  been 
developed  and  analyzed  (Borison,  2005;  Brealey  &  Meyers,  2000;  Dixit  & 
Pindyck,  1994;  Kulatilaka,  1995;  Lander,  1997;  Lander  &  Pinches,  1998; 
McDonald,  2006;  Quigg,  1993;  Teisberg,  1995;  Trigeorgis,  1993, 1995, 

2005)  and  used  to  value  strategies  in  many  domains,  including  specific 
aspects  of  product  development  (Amram  &  Kulatilaka,  1999a;  Brennan  & 
Trigeorgis,  2000;  Dixit  &  Pindyck,  1994;  Kemna,  1993;  Miller  &  Lessard, 
2000;  Trigeorgis,  1995).  Examples  include  valuation  of  options  to  hedge 
technology  investment  risk  (Benaroch,  2001),  and  application  to  design 
modularity  (Baldwin  &  Clark,  2000),  research  and  development  resource 
allocation  (Sharpe  &  Keelin,  1998),  and  maximum  price  contracts  for 
construction  project  options  (Bounkendour  &  Bah,  2001).  Real  Options 
Valuation  Models  have  been  effectively  used  to  demonstrate  how  real 
options  can  increase  project  value,  including  through  engineering  design 
(Baldwin  &  Clark,  2000;  Ford,  Lander,  &  Voyer,  2002;  Park  &  Herath, 
2000;  Zhao  &  Tseng,  2003),  testing,  and  learning  through  pilot  projects 
(Benaroch,  2001;  Sadowsky,  2005),  schedule  control  (Ford  &  Bhargav, 

2006) ,  and  financing  (Cheah  &  Garvin,  2008;  Ho  &  Liu,  2002).  Other 
research  has  demonstrated  the  application  of  real  options,  for  example, 
to  natural  resources  and  land  development;  flexible  manufacturing; 
research,  development,  and  innovation;  mergers  and  acquisitions;  leases; 
and  the  labor  force. 
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Much  of  the  formal  modeling  of  real  options  has  focused  on  valuing 
(in  monetary  terms)  options  for  specific  asset  characteristics  (e.g.,  value, 
uncertainty)  and  option  designs  (timing,  exercise  trigger  conditions,  and 
exercise  costs).  But  researchers  have  identified  common  modeling  assump¬ 
tions  that  do  not  apply  to  typical  product  development  projects  (Lander  & 
Pinches,  1998).  Specifically,  most  Real  Options  Valuation  Models  assume 
that:  (a)  future  asset  behavior  and  value  conform  to  well-defined  processes, 
(b)  markets  are  complete  and  arbitrage  opportunities  are  available,  (c) 
sources  of  uncertainty  are  few  and  independent,  (d)  payouts  or  other  costs 
of  delaying  decisions  are  small,  and  (e)  planners  have  one  or  few  options. 
None  of  these  assumptions  hold  well  for  product  development  environ¬ 
ments,  where  asset  value  behaviors  are  not  well-defined  or  market-based, 
many  sources  of  uncertainty  exist  and  interact,  delaying  decisions  can  be 
very  costly,  and  planners  usually  have,  practically  speaking,  unlimited 
options.  In  addition,  Alessandri,  Ford,  Lander,  Leggio,  and  Taylor  (1999) 
describe  problems  posed  by  assuming  that  asset  performance  and  option 
holder  activity  are  independent,  when  in  fact  product  development  option 
holders  (i.e.,  project  managers)  purposefully  and  significantly  manipulate 
the  linkages  between  uncertainties  and  project  values. 
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In  contrast  to  a  focus  on  option  valuation,  managerial  real  options 
work  to  improve  decision  making  by  structuring  risky  circumstances 
faced  by  practitioners  into  real  options  and  facilitating  option  design  and 
implementation.  Managerial  real  options  address  many  of  the  challenges  of 
using  Real  Options  Valuation  Models  to  improve  risk  management  (Garvin 
&  Ford,  2012;  Triantis,  2005).  Structuring  risk  management  challenges  in 
development  programs  as  real  options  requires  describing  challenges  with 
standard  real  options  parameters  and  structures  (Miller  &  Lessard,  2000). 
Thereby,  real  options  can  improve  managerial  understanding  of  risks  and 
help  managers  prepare  for  risk  management  strategy  design  (Amram  & 
Kulatilaka,  1999a,  1999b;  Bierman  &  Smidt,  1992;  Courtney,  et  al.,  1997; 
Kensinger,  1988).  Potential  benefits  of  real  options  applied  to  product  devel¬ 
opment  stem  from  several  sources,  including:  a  broader  range  of  strategies 
considered,  a  focus  on  objectives  instead  of  solutions,  sensitivity  to  multiple 
project  futures,  more  frequent  testing  of  plans,  and  increased  awareness  of 
the  value  of  flexibility  (Ford,  Lander,  &  Voyer,  2004). 

Consider  the  example  of  the  use  of  options  to  manage  technology 
development  risk  for  the  Department  of  Energy's  $2.4  billion  National 
Ignition  Facility  (Ceylan  &  Ford,  2002).  The  National  Ignition  Facility 
(NIF)  needed  to  develop  slabs  of  laser  glass  to  be  used  in  the  testing  of 
nuclear  weapons  and  research.  Laser  glass  procurement  required  the 
production  of  high-quality  glass  slabs  called  “blanks."  No  existing  glass 
production  technologies  could  produce  the  volume  of  glass  blanks  needed. 
The  ability  of  glass  Arms  to  develop  the  required  new  technologies  and 
the  quality  of  the  glass  produced  if  the  production  technologies  were 
feasible,  were  very  uncertain  as  were  costs  and  development  schedules. 
Although  NIF  had  relationships  with  experienced  laser  glass  vendors, 
none  could  guarantee  successful  development  within  the  required  time  a 
priori.  Therefore,  the  NIF  program  managers  funded  the  development  of 
new  laser  glass  production  technology.  NIF  needed  a  higher  likelihood  of 
success  than  any  one  vendor  could  provide,  so  it  chose  to  hire  two  Arms  to 
simultaneously  begin  initial  development  of  a  technology  to  produce  the 
blanks  and  to  fund  these  development  efforts  in  phases.  Program  manag¬ 
ers  explicitly  incorporated  the  flexibility  to  choose  at  several  stages  to: 

(a)  continue  funding  both  companies  and  their  technology  development, 

(b)  fund  only  one  company  going  forward,  or  (c)  discontinue  funding 
both  companies  and  explore  alternative  sources  for  the  blank  develop¬ 
ment.  The  choice  at  each  stage  was  based  on  what  the  managers  had 
learned  about  the  likelihood  of  success  of  the  developing  Arms  in  meeting 
expectations  (benefits)  and  the  cost  of  continuing  to  fund  the  research 
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and  development.  Through  real  options  analysis  (but  not  formal  option 
valuation),  NIF  managers  were  able  to  assess  the  cost  effectiveness  of 
their  options  at  each  stage.  This  analysis  assisted  managers  in  project 
management  decision  making  by  providing  a  reliable  decision  tool.  See 
Alessandri,  et  al.  (2004),  Cao,  Ford,  and  Leggio  (2006),  and  Ceylan  and 
Ford  (2002)  for  more  detail  and  an  analysis  of  the  use  of  real  options  in 
managing  risk  at  NIF.  The  NIF  example  illustrates  the  critical  role  of 
managerial  decision  making  in  the  application  of  real  options  to  risk 
management  in  development  programs. 


Specific  and  required  decisions  in  the  management 
of  risk  in  an  actual  development  program  created 
the  need  and  opportunity  for  real  options. 


The  current  work  integrates  the  managerial  and  valuation  aspects 
of  real  options.  Specific  and  required  decisions  in  the  management  of 
risk  in  an  actual  development  program  created  the  need  and  opportunity 
for  real  options.  A  simple  and  intuitive  approach  is  used  to  estimate  the 
value  of  options  as  the  difference  between  the  values  of  the  program  with 
and  without  the  options  (e.g.,  by  assuming  uncertainty  impacts  future 
performance  versus  assuming  a  single  specific  and  known  future).  The 
valuations  are  used  to  recommend  managerial  actions  and  improve  the 
understanding  of  the  risk  and  its  management. 

Javelin  Technology  Options 

Anti-tank  weapons  have  been  important  to  meeting  Department 
of  Defense  objectives  since  the  appearance  of  armored  vehicles  on  the 
modern  battlefield  in  World  War  I.  From  the  1960s  through  1970s,  the 
M67  90mm  recoilless  rifle  was  used  as  a  primary  mounted  and  dis¬ 
mounted  infantry  weapon  against  tanks  and  armored  personnel  carriers. 
This  weapon  was  replaced  by  the  DRAGON  anti-tank  weapon  system, 
introduced  in  the  late  1970s,  which  had  a  wired  command  link  that  was 
employed  to  guide  the  missile  to  a  target  that  was  optically  tracked  by  the 
gunner.  The  DRAGON  system  was  developed  in  the  1960s  as  a  response 
to  the  Soviet  development  of  the  AT-3  SAGGER  manpack  missile  system, 
carried  in  a  fiberglass  container  about  the  size  of  a  small  suitcase.  But 
the  DRAGON  system  had  reliability  problems,  and  limited  range  and 
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lethality,  and  it  was  difficult  for  gunners  to  aim  the  missile  and  track 
the  target.  In  the  1980s,  the  goal  was  to  replace  DRAGON  with  a  weapon 
with  increased  range  and  lethality  and  less  weight  (a  later  requirement 
emerged  for  the  ability  to  be  launched  from  inside  an  enclosure  [e.g., 
buildings  and  bunkers]).  The  Advanced  Anti-Armor  Weapon  System- 
Medium  (AAWS-M)  project  grew  into  the  Javelin  anti-tank  missile 
program.  Javelin  required  several  emergent  technologies  in  order  to 
successfully  attain  program  requirements.  A  number  of  the  subsystems 
were  based  upon  these  immature  technologies.  The  target  locating  and 
missile  guidance  subsystems  were  particularly  troublesome  technical 
issues.  Three  very  different  technologies— a  laser-beam  riding  (LBR) 
system,  a  fiber-optic  (FO)  guided  system,  and  the  forward  looking  infra¬ 
red  (FLIR)  system— were  initially  considered  in  a  27-month  technology 
development  phase,  aimed  at  not  constraining  the  eventual  materiel 
solution.  Each  of  the  three  potential  technologies  generally  offered  the 
needed  capabilities  and  represented  acquisition  options. 


The  Advanced  Anti-Armor  Weapon  System- 
Medium  (AAWS-M)  project  grew  into  the  Javelin 
anti-tank  missile  program. 


Three  contractor  teams  were  formed  to  develop  competing  proto¬ 
type  guidance  technologies  for  the  Javelin.  Only  one  team  was  to  be 
chosen  for  follow-on  advanced  development  and  then  production.  Ford 
Aerospace  was  teamed  with  its  partner,  Loral  Systems,  offering  the  LBR 
missile.  Hughes  Aircraft  was  teamed  with  Boeing,  offering  an  FO  guided 
missile.  Texas  Instruments  was  teamed  with  Martin  Marietta,  offer¬ 
ing  an  imaging  infrared  or  FLIR  missile  system.  With  the  LBR  system, 
the  gunner  identifies  the  target  visually  and  points  a  laser  beam  at  the 
target  throughout  flight.  After  launch,  the  missile  continuously  corrects 
its  flight  to  match  the  line  of  the  laser  (to  “ride”  the  laser  beam)  to  the 
target.  The  FO  system  includes  a  coil  of  very  long  and  fine  optical  fiber 
that  connects  the  launch  unit,  operated  by  the  gunner,  to  a  camera  in 
the  nose  of  the  missile.  The  gunner  flies  the  missile  to  the  target  using 
a  joystick  controller  device.  The  FLIR  scans  the  view  in  front  of  the 
gunner  and  generates  a  thermal-based  image  of  the  target  area.  Once 
observed  through  the  Command  Launch  Unit,  or  thermal  sight,  the 
gunner  switches  to  a  staring  array  in  the  missile  to  acquire  the  target 
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by  narrowing  brackets  in  the  viewfinder  around  the  target  with  a  thumb 
switch.  After  launch,  the  missile  continuously  corrects  its  flight  path 
using  a  tracking  algorithm  that  employs  optical  correlators  oriented 
upon  visible  and  distinct  target  features. 

Each  candidate  system  had  specific  advantages  and  disadvantages: 

•  The  Ford/Loral  LBR  required  an  exposed  gunner  and  man- 
in-loop  throughout  its  rapid  flight.  It  was  cheapest  at  an 
estimated  $90,000  cost  per  kill— a  figure  that  was  com¬ 
prised  not  only  of  average  unit  production  cost  estimates, 
but  also  reliability  and  accuracy  estimates.  It  was  fairly 
effective  in  terms  of  potential  combat  utility,  with  dimin¬ 
ishing  probability-of-hit  at  increasing  range.  Top-attack 
on  armor  would  be  dependent  upon  precision  fusing  and 
detonation,  and  accuracy  of  downward-firing  explosively 
formed  projectiles  from  shaped  charges. 

•  The  Hughes/Boeing  FO  guide  prototype  enabled  an 
unexposed  gunner  (once  launched)  and  also  required  man- 
in-loop  throughout  its  slower  flight.  It  was  judged  as  likely 
costlier,  but  less  affected  by  accuracy  throughout  range 
with  its  automatic  lock  and  guidance  in  its  terminal  stage  of 
flight,  and  offered  target  switching.  It  was  also  more  gunner 
training-  and  learning-intensive,  but  could  attack  targets 
from  above,  where  the  armor  was  thinnest. 

•  The  FLIR  prototype  offered  completely  autonomous  fire- 
and-forget  flight  to  target  after  launch,  but  was  perceived 
as  both  the  costliest  and  the  most  technologically  risky 
alternative.  The  gunner  would  only  be  initially  exposed 
during  target  engagement.  It  would  be  the  easiest  system 
upon  which  to  deliver  user  training  and  effective  to  maxi¬ 
mum  ranges  by  means  of  its  target  acquisition  sensor  and 
guidance  packages.  It  used  top  attack  as  a  more  effective 
means  of  armored  target  defeat,  but  would  also  have  a  fiat 
trajectory  capability  for  direct  fire  against  targets  under 
cover  of  bridges,  trees,  buildings,  etc. 
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Effectiveness  of  Technology  Options 

To  choose  a  technology  option,  the  Javelin  development  team  per¬ 
formed  an  analysis  of  alternatives  based  on  the  relative  benefits  and 
costs  of  each  alternative,  (i.e.,  each  alternative's  cost  effectiveness). 
Cost-effectiveness  analysis  is  appropriate  whenever  dollar  values  cannot 


TABLE  1.  ANTI-TANK  MISSILE  GUIDANCE  SYSTEM  EFFECTIVENESS 


Objective 

Measure  Weight 

LBR 

Value  Score 

FO 

Value  Score 

FUR 

Value  Score 

Lethality 

0.3 

P(H)*P(K)** 

0.7 

5 

1.05 

4 

0.84 

7 

1.47 

Top  attack 

0.3 

6 

0.54 

7 

0.63 

9 

0.81 

Total 

1.0 

Tactical 

Advantage 

0.3 

Weight 

0.4 

9 

1.08 

5 

0.6 

3 

0.36 

Time  to  engage 

0.3 

8 

0.72 

7 

0.63 

5 

0.45 

Time  of  flight 

0.2 

7 

0.42 

5 

0.3 

5 

0.3 

Redirect 

capability 

0.1 

10 

0.3 

10 

0.3 

0 

0 

Total 

1.0 

Gunner  Safety 

0.3 

Required 

training 

0.2 

5 

0.3 

1 

0.6 

10 

0.6 

Exposure  after 
launch 

0.8 

2 

0.48 

8 

1.92 

10 

2.4 

Total 

1.0 

Procurement 

0.1 

Ease  of 
procurement 

1.0 

8 

0.8 

6 

0.6 

4 

0.4 

Total 

1.0 

MOE  (Total)*** 

1.0 

5.69 

5.88 

6.79 

*P(H)  =  Probability  of  hit 
**P(K)  =  Probability  of  kill 
***MOE  =  Measure  of  effectiveness 
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be  assigned  to  benefits,  as  is  the  case  for  most  defense  systems  (Office 
of  Management  and  Budget,  1992).  To  evaluate  the  effectiveness  of 
each  Javelin  guidance  system  alternative,  we  use  a  simple  hierarchical 
model  based  on  the  acquisition  objectives  identified  for  the  anti-tank 
missile— lethality,  tactical  advantage,  gunner  safety,  and  procurement.^ 
Our  multiobjective  effectiveness  model  is  based  on  concepts  developed 
in  decision  science  (Buede,  1986;  Keeney,  1982;  Keeney,  1988).  The  first 
three  objectives  deal  with  the  operational  effectiveness  of  the  missile, 
while  the  procurement  objective  recognizes  that  transaction  costs  and 
technology  issues  make  some  alternatives  easier  to  procure  than  others. 
Under  each  objective,  we  can  use  metrics  to  measure  how  well  the  objec¬ 
tive  is  achieved.  The  objectives  and  corresponding  metrics  (measures) 
are  shown  in  Table  1.  The  table  also  shows  the  relative  importance  of 
objectives  and  the  relative  importance  of  each  measure  with  respect  to 
an  objective  by  the  weight  assigned  to  each  objective  and  measure. 

The  metric  values  achieved  by  each  technology  are  converted  to  a 
notional  value  measured  on  a  scale  of  0  to  10,  indicating  the  value  the 
Army  assigned  to  the  actual  level  of  performance.  A  notional  measure  of 
effectiveness  (MOE)  achieved  by  each  of  the  three  alternatives  is  shown  at 
the  bottom  of  Table  1.  The  scores  shown  in  Table  1  for  each  metric  (mea¬ 
sure)  are  calculated  by  multiplying  the  value  for  the  measure  times  the 
weight  for  that  measure  times  the  objective  weight.  The  notional  MOEs 
in  Table  1  are  consistent  with  the  Army's  Source  Selection  Evaluation 
Board  (SSEB)  preference  for  the  three  guidance  technologies,  in  that  the 
SSEB  preferred  FLIR  over  the  other  two  guidance  systems  and  perceived 
the  FO  system  as  being  slightly  better  than  the  LBR  system. 

Cost-Effectiveness  Analysis 

The  previous  section  considered  only  the  relative  effectiveness  of  the 
three  guidance  alternatives.  To  select  the  best  alternative,  we  must  also 
consider  the  development  and  procurement  costs.  Table  2  shows  notional 
values  for  the  cost  per  missile  for  each  alternative  and  the  total  program 


TABLE  2.  ANTI-TANK  MISSILE  COST 

LBR 

FO 

FLIR 

Cost/missile  ($M) 

$0.09 

$0.11 

$0.15 

Program  Cost  ($M) 

$180 

$220 

$300 
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cost  for  each  alternative  assuming  2,000  missiles  are  procured.  A  cost 
versus  effectiveness  graph  is  shown  in  Figure  1.  The  total  program  cost 
and  MOE  for  each  alternative  are  shown  on  the  graph. 

Figure  1  illustrates  that  no  alternative  dominates  another,  meaning 
no  alternative  is  both  cheaper  and  more  effective  than  another.  Thus, 
we  must  look  at  the  marginal  benefit  and  marginal  cost  to  evaluate  the 
alternatives.  The  LBR  alternative  is  the  least  costly  and  least  effective.  We 
compare  it  to  the  FO  alternative  in  Table  3  and  note  that  the  marginal  cost 
of  choosing  FO  over  LBR  is  $40  million.  Table  3  also  shows  the  difference 
in  values  for  each  of  the  effectiveness  measures  used  to  calculate  the  MOE 
between  the  two  alternatives.  A  positive  change  represents  an  increase  in 
effectiveness,  while  a  negative  difference  indicates  a  decrease  in  effective¬ 
ness.  A  similar  analysis  for  FO  versus  FLIR  is  shown  in  Table  4. 

While  Figure  1  gives  us  an  overall  picture  of  the  cost  versus  effective¬ 
ness  of  the  three  alternatives.  Tables  3  and  4  show  what  is  gained  and  lost 
at  the  margin  when  going  from  one  technology  to  the  next.  Arguably,  this 
is  captured  in  the  overall  MOE,  but  decision  makers  are  often  interested 

FIGURE  1.  ANTI-TANK  MISSILE  TOTAL  PROGRAM  COST  VS. 

EFFECTIVENESS 
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in  seeing  what  specifically  they  are  getting  for  their  investment.  In  addi¬ 
tion,  because  the  MOE  is  a  combination  of  different  metrics  that  are  not 
necessarily  interchangeable,  it  would  not  make  sense  to  simply  calculate 
the  ratio  of  MOE  to  cost.  Instead,  the  marginal  analysis  defines  the  trade¬ 
off  space,  but  not  the  solution,  for  the  decision  makers. 

The  previous  cost  versus  effectiveness  analysis  assumed  that  all  the 
guidance  technology  development  efforts  would  be  equally  successful 
and  achieve  the  calculated  MOEs.  But  at  the  start  of  the  proof-of-prin- 
ciple  effort,  there  was  no  assurance  that  any  of  the  technologies  would 
be  successfully  developed.  In  fact,  the  probability  of  success  differed 
between  the  three  alternatives.  A  notional  assessment  of  the  probability 
of  success^  for  each  option  is  given  in  Table  5. 

Table  5  also  shows  the  expected  MOE  based  on  the  given  probability 
of  success  for  each  option.  The  expected  MOE  is  the  product  of  the  MOE 
shown  in  Table  1  and  the  probability  of  success. 

Table  5  incorporates  uncertainty  into  the  estimate  of  the  benefits 
(as  measured  by  the  MOE)  of  each  alternative.  However,  it  does  not 
incorporate  uncertainty  into  the  costs  of  those  alternatives.  Each  of  the 
alternatives  has  an  expected  cost  based  on  the  uncertainty  associated 
with  the  technology  development.  If  the  development  effort  fails,  we 
assume  the  Army  will  have  to  pay  some  additional  cost  to  finish  devel¬ 
oping  the  technology  and  achieve  the  anticipated  level  of  effectiveness 
(MOE).  We  refer  to  this  additional  cost  as  the  “cost  to  fix”  the  technol¬ 
ogy.  If  the  technology  development  phase  is  successful,  then  the  cost  to 
exercise  the  option  will  be  the  total  program  cost  from  Table  2  and  is 
shown  as  “cost  to  implement”  in  Table  6.  If  the  technology  development 
phase  fails,  the  cost  of  the  alternative  will  be  the  cost  to  implement  plus 
the  cost  to  fix  the  technology.  The  total  expected  cost  of  each  alternative 
is  given  in  Table  6. 

The  values  shown  in  Table  6  assume  that  one  of  the  technologies  will 
be  picked  based  on  the  cost-versus-benefit  analysis  presented  above  and 
that,  whichever  technology  is  chosen,  there  is  an  additional  cost  to  achieve 
the  anticipated  effectiveness  (MOE)  if  the  development  phase  fails.  Figure 
2  incorporates  uncertainty  into  our  analysis,  showing  expected  cost  (from 
Table  6)  versus  expected  effectiveness  (from  Table  5). 
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TABLE  3.  MARGINAL  ANALYSIS  OF  COST  AND  EFFECTIVENESS 
FOR  LBR  AND  FO 


Marginal  Analysis 

LBR 

FO 

Difference 

Program  Cost  ($M) 

$180 

$220 

$40 

P(H)*P(K) 

5 

4 

-1 

Top  attack 

6 

7 

+  1 

Weight 

9 

5 

-4 

Time  to  engage 

8 

7 

-1 

Time  of  flight 

7 

5 

-2 

Redirect  capability 

10 

10 

0 

Required  training 

5 

1 

-4 

Exposure  after  launch 

2 

8 

+6 

Ease  of  procurement 

8 

6 

-2 

TABLE  4.  MARGINAL  ANALYSIS  OF  COST  AND  EFFECTIVENESS 
FOR  FO  AND  FLIR 


Marginal  Analysis 

FO 

FUR 

Difference 

Program  Cost  ($M) 

$220 

$300 

$80 

P(H)*P(K) 

4 

7 

+3 

Top  attack 

7 

9 

+2 

Weight 

5 

3 

-2 

Time  to  engage 

7 

5 

-2 

Time  of  flight 

5 

5 

0 

Recall  capability 

10 

0 

-10 

Required  training 

1 

10 

+9 

Exposure  after  launch 

8 

10 

+2 

Ease  of  procurement 

6 

4 

-2 
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TABLE  5.  PROBABILITY  OF  DEVELOPMENT  SUCCESS  AND 
EXPECTED  MOE  FOR  JAVELIN  TECHNOLOGY  OPTIONS 


LBR 

FO 

FUR 

Probability  of  success 

0.6 

0.5 

0.4 

Expected  measure  of  _  .  _ 

effectiveness 

3.414 

2.94 

2.716 

TABLE  6.  EXPECTED  COST  OF  JAVELIN  GUIDANCE  TECHNOLOGY 
ALTERNATIVES  WITHOUT  OPTION  TO  TERMINATE  PROJECT 


LBR 

FO 

FUR 

Probability  of  success 

0.6 

0.5 

0.4 

Cost  to  implement  ($M) 

180 

220 

300 

Probability  of  failure 

0.4 

0.5 

0.6 

Cost  to  fix 

50 

70 

90 

Total  cost  to  implement 
given  failure  ($M) 

230 

290 

390 

Expected  Cost  ($M) 

200 

255 

354 

TABLE  7.  VALUE  OF  JAVELIN  GUIDANCE  TECHNOLOGY  OPTIONS 


LBR 

FO 

FLIR 

Probability  of  success 

0.6 

0.5 

0.4 

Cost  to  implement 

180 

220 

300 

Probability  of  failure 

0.4 

0.5 

0.6 

Cost  if  project  is  terminated 

0 

0 

0 

Expected  cost  with  option 

108 

110 

120 

Expected  cost  w/o  option 

200 

255 

354 

Value  of  option 

92 

145 

234 
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Using  the  expected  cost  and  expected  MOE,  we  see  that  both  the  FO 
and  FLIR  guidance  systems  are  dominated  by  the  LBR  system,  since  it 
is  both  cheaper  and  has  a  higher  MOE.  This  analysis  suggests  that,  based 
on  the  information  available  at  the  beginning  of  the  program,  the  LBR 
technology  should  be  chosen.^ 

Value  of  Technology  Options 

A  real-options  approach  differs  from  the  choose-early  approach  above 
in  that  it  purposely  delays  making  a  decision  while  more  or  better  informa¬ 
tion  is  gathered.  The  value  of  a  real  option  is  derived  from  the  difference 
between  the  expected  net  value  (benefits  less  costs)  of  an  investment  and 
the  net  value  of  that  investment  given  that  it  succeeds.  In  the  Javelin  case, 
the  option  value  lies  in  the  flexibility  to  terminate  the  development  of  a  tech¬ 
nology  if  it  is  not  successful.^  To  develop  a  simple  option  valuation  model 
for  the  Javelin  guidance  technologies,  we  note  that  the  benefits  are  given 
by  the  MOE  as  shown  in  Table  1.  If  we  do  not  use  a  real-options  approach, 
then  the  costs  of  each  alternative  are  shown  in  Table  6. 


Using  a  real-options  approach,  the  Army  pays  for  the  option  to  And 
out  whether  the  technology  development  succeeds  before  making  a  final 
choice.  If  the  development  succeeds,  we  have  achieved  the  MOE  shown  in 


FIGURE  2.  ANTI-TANK  MISSILE  TOTAL  EXPECTED  PROGRAM 
COST  VS.  EXPECTED  EFFECTIVENESS 
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Table  1  and  can  proceed  with  the  project  if  we  prefer  that  option  (based 
on  the  cost  versus  effectiveness  analysis  presented  in  the  previous  sec¬ 
tion).  If  the  development  fails,  we  terminate  the  project  and  there  is  no 
further  cost.  The  value  of  the  option  is  given  by  the  difference  between 
the  expected  value  of  the  project  with  no  option  (from  Table  6)  and  the 
expected  value  of  the  project  with  the  option  to  terminate.  The  calcula¬ 
tions  are  shown  in  Table  7.^ 

The  values  of  the  options  for  each  alternative  are  different  because 
different  levels  of  uncertainty  are  associated  with  each  technology.  For 
example,  uncertainty  on  the  two  relatively  mature  technologies  (LBR  and 
FO)  was  lower  than  for  the  new  technology  (FLIR).  Given  that  we  are  willing 
to  pay  for  the  technology  with  no  options,  the  more  uncertain  the  technol¬ 
ogy,  the  more  we  value  the  option  to  terminate  the  project  if  the  technology 
development  fails.  The  values  shown  in  Table  7  are  maximums  in  the  sense 
that  if  we  pay  any  more  than  the  option  value,  we  would  have  been  better  off 
not  using  an  option.  If  we  pay  less  than  the  option  value,  we  experience  real 
cost  savings  by  not  expending  funds  on  an  unsuccessful  technology. 

Suppose  the  Army  preferred  the  LBR  technology  (based  on  the  cost 
versus  effectiveness  analysis  presented  in  the  previous  section).  They 
should  pay  no  more  than  $92  million  for  the  option  to  terminate  the  proj¬ 
ect.  But  the  Army  decided  to  buy  options  for  all  three  technologies,  so  the 
total  amount  that  they  spend  on  options  should  not  exceed  the  value  of 
the  option  for  the  preferred  technology.  Since  the  value  of  the  LBR  option 
is  $92  million,  if  they  allocated  the  option  value  equally  across  all  the 
alternatives,  they  should  spend  no  more  than  about  $30  million  for  each 
option,  which  is  exactly  what  the  Army  did. 

But  allocating  the  option  value  equally  across  the  alternatives  is  not 
economically  optimized,  given  that  some  technologies  are  more  uncer¬ 
tain  than  others.  It  would  be  more  rational  to  allocate  the  option  value 
based  on  the  level  of  uncertainty  that  we  are  trying  to  resolve.  Using  the 
probability  of  failure  in  Table  7  as  a  notional  measure  of  risk,  we  can 
allocate  the  option  value  in  proportion  to  that  risk,  giving  27  percent  of 
the  total  option  value  to  LBR,  33  percent  to  FO,  and  40  percent  to  FLIR. 
Going  back  to  our  previous  example,  if  the  Army  prefers  the  LBR  tech¬ 
nology,  then  the  total  cost  of  the  option  should  not  exceed  $92  million. 
That  means  the  Army  should  pay  about  $25  million  for  the  LBR  option, 
$30  million  for  the  FO  option,  and  $37  million  for  the  FLIR  option.  Doing 
so  allocates  the  dollars  based  on  risk  while  keeping  the  total  cost  equal 
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to  the  option  value  of  the  preferred  alternative.  Again,  we  note  that  $92 
million  is  a  maximum.  To  realize  any  cost  savings  from  the  option,  the 
Army  must  pay  less  than  $92  million  for  all  three  options. 

The  Army’s  Choice 

The  Army's  Cost  and  Operational  Effectiveness  Analysis  (COEA)  indi¬ 
cated  that  the  LBR  was  the  preferred  alternative  based  on  weighted  cost/ 
efficiency  factors.  At  the  same  time,  the  SSEB  actually  chose  the  FLIR  alter¬ 
native  because  of  a  bias  toward  fire  and  forget.  This  difference  illustrates 
the  importance  of  choosing  decision  criteria  that  accurately  reflect  the 
needs  and  preferences  of  warfighters  for  analysis  of  alternatives.  Although 
time  of  flight  and  gunner  survivability  were  not  stated  requirements  in  the 
AAWS-M  Joint  Required  Operational  Capability  document  per  se,  fire 
and  forget  nevertheless  translated  into  greatly  enhanced  gunner  surviv¬ 
ability  and  overwhelmingly  appealed  to  user  representatives.  The  resulting 
decision  by  the  SSEB  reflected  this  preference.  In  June  1989,  a  full-scale 
development  (now  called  Engineering  and  Manufacturing  Development,  or 
EMD)  contract  was  awarded  for  the  AAWS-M  project  to  the  joint  venture 
team  of  Texas  Instruments  and  Martin  Marietta.  At  the  macro  level,  the 
Office  of  the  Secretary  of  Defense  viewed  the  program  as  acceptable  regard¬ 
ing  risk  because  of  its  27-month  technology  development  phase,  use  of 
multiple  prototypes  (real  options)  for  a  technical  solution,  and  subsequent 
36 -month  plan  for  full-scale  development.  But  at  the  program  office  level, 
it  was  known  to  be  one  of  high  risk  in  several  technical  areas.  Focal  Plane 
Array  (FPA)  technology  was  still  immature  and  would  be  gauged  today 
at  approximately  Technology  Readiness  Level  (TRL)  5  (on  an  increasing 
scale  of  1-9),  despite  its  successful  technology-development  phase  results. 

About  18  months  into  the  EMD  phase,  serious  technical  problems 
around  FPA  attainment  of  specified  sensitivity  and  production  yield, 
system  weight,  tracker  algorithm,  and  other  areas  doubled  the  expected 
cost  of  development  and  added  about  18  more  months  to  the  originally 
planned  36  months  to  complete.  Over  that  next  year,  the  program  sought 
a  new  baseline  with  many  different  revised  program  estimates— climbing 
from  36  months'  duration  and  $298  million  in  cost,  to  48  months'  dura¬ 
tion  and  $372  million  in  cost,  and  finally,  54  months  and  $443  million  for 
the  total  cost  and  duration  of  this  phase.  Within  that  year,  the  program 
was  restructured,  given  the  new  baseline,  and  finished  largely  within  its 
new  parameters.  The  additional  18  months  added  to  the  36 -month  phase 
helped  resolve  the  uncertainties  and  complexities  of  system  development 
without  additional  schedule  slippage. 
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Today,  Javelin  is  viewed  as  being  a  totally  successful  weapon  system, 
despite  its  much  earlier  programmatic  shortcomings  in  development.  It 
is  being  used  in  combat  operations  and  has  continued  through  many  full- 
rate  production  contract  periods.  Over  1,000  Javelin  missiles  have  been 
fired  in  Operations  Iraqi  Freedom  and  Enduring  Freedom  since  March 
2003,  with  close  to  98  percent  reliability.  The  system  design  has  con¬ 
tinued  to  be  upgraded— not  as  blocks  of  capability,  but  with  incremental 
software,  warhead,  and  producibility  enhancements. 


Javelin  missiles  have  been  fired  in  Operations  Iraqi 
Freedom  and  Enduring  Freedom  since  March  2003^ 
with  close  to  98  percent  reliability. 


Conclusions 

Several  observations  can  be  made  from  our  analysis  of  the  Javelin 
guidance  technology  acquisition  process. 

The  first  is  that  the  benefit  of  weapon  systems,  or  in  this  case  mis¬ 
sile  guidance  systems,  is  not  measured  in  dollars.  This  makes  using  a 
traditional  option  valuation  model  (based  on  benefits  less  costs)  difficult. 
Instead,  we  must  use  the  principles  of  multiobjective  decision  making  to 
develop  an  MOE  for  each  alternative.  The  MOE  can  be  compared  to  the 
cost  to  define  the  trade  space  for  the  decision  maker. 

While  the  MOEs  calculated  in  this  article  (Table  1)  aligned  with  the 
preferences  of  the  SSEB,  which  were  heavily  weighted  towards  gunner 
safety,  it  is  possible  that  the  intial  Army  COEA  did  not  adequately  con¬ 
sider  what  turned  out  to  be  the  most  important  objective— gunner  safety. 
If  we  remove  that  objective  from  Table  1  and  assign  equal  weights  of  0.45 
to  Lethality  and  Tactical  Advantage,  leaving  Procurement  with  a  weight 
of  0.1,  the  revised  MOEs  would  yield  a  result  similar  to  the  Army's  COEA, 
with  LBR  receiving  the  highest  MOE.  However,  it  is  also  possible  to  select 
FLIR  as  the  preferred  alternative  based  on  Figure  1  (cost-effectiveness) 
if  the  SSEB  considered  effectiveness  to  be  much  more  important  than 
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cost,  basically  concluding  that  the  FLIR  was  worth  the  extra  money.  Both 
cases  illustrate  the  subjective  nature  of  effectiveness  analysis  and  why  it 
is  so  important  to  explicitly  model  objectives  and  preferences. 

Second,  we  note  that  the  three  proposed  guidance  technologies  had 
different  levels  of  risk.  We  used  this  information  to  calculate  the  expected 
MOE  for  each  alternative,  thus  incorporating  uncertainty  into  the  analysis. 
The  probabilistic  MOE  can  be  compared  to  the  expected  cost  (in  our  case, 
cost  per  kill)  to  present  a  risk- adjusted  trade  space  for  the  decision  maker. 
To  the  best  of  our  knowledge,  neither  the  COE  A  nor  the  SSEB  used  a  quan¬ 
titative  risk-adjusted  trade  space  as  presented  in  Figure  2  in  their  analysis, 
although  it  is  likely  that  they  considered  technology  risk  qualitatively.  The 
risk-adjusted  model  presented  in  this  article  would  have  provided  a  clearer 
picture  of  the  impact  of  technology  uncertainty  on  the  trade  space. 

Third,  we  show  that  a  real-options  approach  allows  us  not  only  to 
incorporate  uncertainty  in  our  analysis,  but  also  to  calculate  the  value  of 
the  option  based  upon  risk.  This  leads  to  different  option  values  for  differ¬ 
ent  alternatives  based  on  the  technology  maturity.  Using  this  approach, 
the  Army  should  have  offered  each  development  team  a  different  amount 
of  money  to  develop  their  proposed  technology.  Doing  so  would  have  better 
allocated  the  dollars  to  manage  risk. 

Fourth,  we  note  that  the  final  cost  to  fix  the  FLIR  guidance  technology 
selected  by  the  Army  turned  out  to  be  significantly  higher  than  the  $30  mil¬ 
lion  originally  paid  to  develop  the  technology.  This  is  in  line  with  what  our 
Real  Options  Valuation  Model  suggested,  since  the  FLIR  technology  was 
always  anticipated  to  be  the  riskiest. 

The  use  of  risk-based  Real  Options  Valuation  Models  allows  us  to  esti¬ 
mate  the  value  of  flexibility  in  acquisition  decisions.  Understanding  this 
value  allows  program  managers  to  assign  program  dollars  based  on  risk  and 
supports  the  efficient  use  of  limited  program  resources.  Program  manag¬ 
ers  can  develop  more  effective  capabilities  more  efficiently  by  improving 
and  expanding  their  understanding  of  the  potential  and  use  of  real  options 
in  acquisition.  Future  research  can  facilitate  these  improvements  by 
investigating  the  role  of  real  options  in  other  development  phases  such  as 
development  for  production,  the  impacts  of  imperfect  information  obtained 
through  options  that  inaccurately  indicate  the  feasibility  of  a  technology, 
and  other  aspects  of  real  options  in  acquisition. 
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Endnotes 

1.  The  objectives,  metrics,  weights  and  values  used  in  this  example  were 

developed  by  the  third  author,  a  program  manager  for  the  Javelin  who 
was  deeply  involved  in  the  Army  evaluation  process.  The  Services 
procuring  the  Javelin  system  did  not  actually  use  this  exact  methodology 
for  the  selection  of  the  Javelin  guidance  technology,  but  used  something 
similar  for  a  weighted  decision  analysis  of  the  three  alternatives. 

2.  The  probabilities  of  success  and  failure  as  well  as  the  costs  for  each 

technology  were  estimated  by  the  third  author  based  on  extensive 
experience  as  a  program  manager  for  the  Javelin. 

3.  The  Army’s  actual  Cost  and  Operational  Effectiveness  Analysis  (COEA) 

also  found  that  the  LBR  was  the  preferred  alternative. 

4.  The  focus  in  the  current  work  is  on  the  technology  development  phase  and 

the  feasibility  of  the  technologies  (e.g.,  as  measured  with  the  TRL),  as 
opposed  to  their  readiness  for  production.  We  assume  that  the  options 
provide  perfect  information  on  the  feasibility  of  the  technology. 

5.  The  expected  cost  with  option  assumes  perfect  information  relative  to  the 

success  of  the  technology  development  effort. 
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Better  Buying  Power  or 
Better  Off  Not? 

Purchasing  Technical  Data 
for  Weapon  Systems 

f  James  Hasik 

In  Septennber  2010,  then-Under  Secretary  of  Defense 
for  Acquisition,  Technology  and  Logistics  Ashton  Carter 
directed  program  managers  to  routinely  analyze  the  busi¬ 
ness  cases  behind  procuring  the  technical  data  packages 
and  rights  to  new  weapon  systems.  In  this  article,  the 
author  recounts  some  of  the  historical  difficulties  with 
procuring  technical  data  for  fielded  systems,  and  presents 
a  heuristic  economic  model  outlining  the  problems  that 
PMs  should  consider  before  making  an  offer. 


<■<'  Image  designed  by  Diane  Fieischer 
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With  tighter  spending  constraints^  continued 
underperformance  could  become  dangerous  on  the 
frontline. 


In  September  2010,  then-Under  Secretary  of  Defense  for  Acquisition, 
Technology  and  Logistics  Ashton  Carter  issued  a  broad  memorandum 
on  acquisition  reform  to  the  heads  of  the  military  services  and  defense 
agencies  (Carter,  2010).  Covering  five  major  themes,  with  23  individual 
initiatives,  but  published  in  just  10  pages,  the  memorandum  called  for  a 
thorough  rethinking  of  how  the  Pentagon  went  about  acquiring  goods 
and  services— an  approach  aimed  at  developing  ''better  buying  power.'’ 
Indeed,  Better  Buying  Power  was  the  document's  chosen  title— a  term 
intended  as  a  quiet  rallying  cry  for  better  performance  in  the  business 
of  defense. 


Background 

Underperformance  and  Shrinking  Budgets 

As  the  largest  buyer  by  far  of  weapons  worldwide,  and  the  monopsony 
buyer  in  the  largest  market,  the  U.S.  Government  ought  to  be  a  more 
powerful  buyer.  Carter  believed,  extracting  better  terms  than  it  had 
historically.  Why  the  sudden  imperative?  As  Carter  told  an  assemblage 
of  industrialists  in  a  progress  report  10  months  later,  their  generation  had 
"grown  accustomed  over  the  post- 9/11  decade  to  circumstances  in  which 
we  could  always  reach  for  more  money."  The  problem,  he  continued,  was 
that  "those  days  are  gone"  (Marshall,  2011).  Amidst  a  financial  crisis  and 
a  nearly  global  recession,  the  military  budget  clearly  would  be  decreasing. 
With  tighter  spending  constraints,  continued  underperformance  could 
become  dangerous  on  the  frontline. 

To  be  fair,  there  had  been  some  remarkable  success  stories  in  the 
wars  in  Afghanistan  and  Iraq  with  rapid,  off-the-shelf  procurements. 
But  many  developmental  programs  had  lurched  from  delay  to  cost  over¬ 
run.  The  plight  of  the  Army  was  remarkably  bad.  Twenty-two  of  the 
Service's  major  weapon  systems  programs  had  been  cancelled  since 
1995,  at  a  cost  of  $32  billion  for  materiel  never  fielded  (Capaccio,  2011). 
Perhaps  this  should  not  have  been  news.  Carter  had  previous  experi¬ 
ence  in  the  Pentagon,  and  was  a  professor  of  public  policy  at  Harvard's 
Kennedy  School  of  Government.  As  such,  he  likely  agreed  with  Asher  and 
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Maggelet's  assertion  of  almost  three  decades  prior  that  “schedule  and 
cost  growth  in  DoD  weapon  systems  acquisition  have  been  recognized 
as  an  economic  fact  of  life”  (1984,  p.  iii).  He  simply  believed  in  breaking 
that  supposition. 


The  dilemma  with  which  Carter  and  his  team  had 
grappled  was  ^^how  to  incentivize  lower  prices  in 
the  short  run  without  ruining  suppliers^  incentives 
to  commit  assets,  incur  risk,  and  innovate  for  the 
long  run/’ 


Aiming  to  do  so,  Carter's  planners  took  in  more  than  one  hundred 
ideas  for  reforms,  and  whittled  the  list  down  to  the  23  they  considered 
“long-ball  hitters”  (McFarland,  2011,  p.  7).  There  were  clear  themes: 
the  word  “incentive”  appears  13  times  in  the  memorandum,  and  the 
word  “competition”  fully  50.  In  particular,  as  a  former  chief  industrial 
strategist  at  the  Pentagon  observed,  “Better  Buying  Power  has  taken 
aim  at  eradicating  what  it  views  as  a  sclerosis  of  comfortable  contractor 
incumbency...  and  reads  like  a  monopsonist's  playbook  for  defense  in  the 
21st  century”  (Grundman,  2010,  p.  3).  The  dilemma  with  which  Carter 
and  his  team  had  grappled  was  “how  to  incentivize  lower  prices  in  the 
short  run  without  ruining  suppliers'  incentives  to  commit  assets,  incur 
risk,  and  innovate  for  the  long  run”  (Grundman,  2010,  pp.  1, 3).  That  long 
run  would  last  a  bit,  for  after  Carter  moved  up  to  become  the  new  deputy 
secretary  of  defense,  incoming  acting  Under  Secretary  Frank  Kendall 
(2011b)  issued  a  two-page  memorandum  largely  staying  the  course  of 
Better  Buying  Power. 

How  easily  the  strategy  would  be  implemented  by  program  managers 
might  be  another  question.  In  the  ensuing  pages,  I  recount  some  of  the 
historical  difficulties  with  procuring  technical  data  for  fielded  systems, 
and  present  a  heuristic  economic  model  outlining  the  problems  that 
program  managers  (PM)  should  consider  before  making  an  offer. 
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The  Technical  Data  Package  Explained 

Competition  amongst  prospective  contractors  is  one  natural 
way  of  inducing  lower  prices.  So,  under  the  heading  "Promote  Real 
Competition,”  and  subheading  "Remove  Obstacles  to  Competition,” 
one  of  those  23  instructions  called  for  the  Services  to  "Require  Open 
Systems  Architectures  and  Set  Rules  for  the  Acquisition  of  Technical 
Data  Rights.”  Specifically,  Dr.  Carter  (2010)  wrote  that: 

At  Milestone  B,  I  will  require  that  a  business  case  analysis  be  con¬ 
ducted  in  concert  with  the  engineering  trades  analysis  that  would 
outline  an  approach  for  using  open  systems  architectures  and 
acquiring  technical  data  rights  (TDRs)  to  ensure  sustained  consid¬ 
eration  of  competition  in  the  acquisition  of  weapon  systems,  (p.  10) 

This  was  not  strictly  a  revolutionary  thought.  Prior  to  Carter's  initia¬ 
tive,  the  John  Warner  Weapon  Systems  Acquisition  Reform  Act  (WSARA) 
of 2009  had  mandated  that  the  acquisition  strategy  for  any  Major  Defense 
Acquisition  Program  (MDAP)  provide  a  plan  to  ensure  at  least  the  com¬ 
petition  throughout  a  system's  life  cycle.  The  WSARA  listed  10  possible 
measures  to  consider;  buying  the  technical  data  package  (TDP)  was  one  of 
those  (Byrd,  2010,  p.  10).  Carter's  memorandum  moved  that  from profeafeZy 
should  to  definitely  should  consider.  The  Army  then  got  its  own  recommen¬ 
dation  from  the  Decker-Wagner  Army  Acquisition  Review  Panel,  which 
recommended  buying  TDPs  during  development,  so  long  as  that  was  "con¬ 
sistent  with  the  risk-reward”  (Decker-Wagner,  2011,  p.  xvi). 

But  what  precisely  are  TDRs  and  TDPs?  While  sometimes  confiated, 
the  terms  are  not  synonymous.  The  Defense  Acquisition  University's 
Glossary  (2005,  p.  B-181)  defines  the  TDP  as  "a  relatively  complete  pack¬ 
age  of  design  and  manufacturing  information”  consisting  of  "drawings, 
quality  assurance  provisions,  standards,  performance  requirements, 
quality  assurance  provisions,  and  packaging  details.”  Depending  on 
the  contract  terms  under  which  a  weapon  system  was  developed,  TDRs 
confer  some  degree  of  legal  authorization  "to  use,  duplicate,  or  disclose” 
those  data,  potentially  to  a  competing  contractor  (p.  B-78).  Thus,  the 
TDP  is  the  actual  intellectual  content  of  the  TDRs,  but  possession  of 
one  is  not  possession  of  the  other.  Frankly,  neither  is  necessary  for 
ensuring  competition  before  a  system  enters  initial  production,  but  both 
figure  strongly  for  ensuring  competition  afterwards.  If  the  government 
lacks  the  data  and  rights  thereto,  the  contractor  that  designed  a  weapon 
system  will  undergo  a  "fundamental  transformation”  from  applicant 
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to  incumbent  (Williamson,  1988,  p.  80),  and  stand  alone  as  sole  source 
for  reorders,  upgrades,  overhauls,  or  possibly  even  spares.  The  govern¬ 
ment  may  still  stand  before  the  contractor  as  a  monopsonist,  as  the 
only  domestic  buyer  of  heavy  weapons,  and  the  veto  authority  on  arms 
exports.  But  even  then,  the  buyer-seller  relationship  would  be  a  bilateral 
monopoly— a  problematic  negotiating  situation.  One  of  the  chief  interests 
in  acquiring  TDPs  and  TDRs  should  thus  be  clear:  with  a  full  understand¬ 
ing  of  how  to  produce  and  maintain  a  system,  the  government  can  open 
a  second  source— a  potential  alternative  to  the  incumbent. 

A  Brief  History  of  the  Government’s  Stance  on  Technicai 
Data 

With  such  benefits,  one  might  presume  that  the  government  has 
always  and  everywhere  wanted  its  data,  but  the  policy  has  varied  over 
time.  Naturally,  the  government  rarely  acquires  technical  rights  to  wholly 
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commercial  items— there  is  simply  no  reason  to  own  the  blueprints  for 
making  readily  available  items  such  as  standard  screws  in-house.  But 
even  with  noncommercial  items  developed  at  governmental  expense, 
through  the  end  of  World  War  II,  both  the  War  and  Navy  Departments 
rarely  acquired  TDRs.  Contractors  were  hardly  willing  to  sign  them  away, 
having  a  natural  interest  in  exclusivity,  as  sole  possession  would  forestall 
competition.  Even  the  government  may  not  want  to  push  too  hard  for 
TDRs.  As  noted  earlier,  Williamson's  “fundamental  transformation”  to 
incumbency  brings  stability  to  the  business  relationship.  The  presump¬ 
tion  of  future  quasi  rents  from  monopolistic  competition  may  encourage 
long-run  innovation,  which  the  monopsonist  must  take  care  not  to  kill,  as 
defense  is  presumably  a  long-run  game  (Grundman,  2010,  p.  1). 

For  decades,  contractors  had  little  to  fear.  Armed  Services 
Procurement  Regulation  No.  9  ensured  that  whatever  technical  data 
the  government  might  acquire  alongside  its  armaments,  it  would  oth¬ 
erwise  not  possess  “any  right  to  reproduce  anything  else  called  for 
by  this  contract”  (McKie,  1966,  p.  5).  Thus,  TDPs  frequently— TDRs 
almost  never.  But  in  1955,  the  escalating  cost  of  new  aircraft  led  the 
Defense  Department  to  assert  that  the  aforementioned  clause  was  not 
so  restrictive,  and  that  the  government's  data  rights  could  be  extended 
to  competing  suppliers  without  royalty.  Faced  with  such  severe  regime 
instability,  quite  a  few  contractors  rebelled.  Over  the  next  10  years,  the 
Pentagon's  technical  data  regulations  underwent  four  revisions,  culmi¬ 
nating  in  a  state  of  considerable  rights  for  contractors  (Maizel,  1986, 
pp.  236-245).  As  those  rights  remained  inadequately  defined,  a  flurry  of 
litigation  ensued,  until  passage  of  the  1983  Defense  Procurement  Reform 
Act  and  the  1984  Competition  in  Contracting  Act  emphasized  assertion 
of  greater  governmental  rights  (Maizel,  1986,  pp.  270-271). 

In  1993,  however,  the  Clinton  administration  entered  office  deter¬ 
mined  to  “reinvent  government”  with  thoroughly  businesslike  practices. 
Buying  suppliers'  technical  data  was  not  (and  still  is  not)  common 
commercial  practice,  so  the  mandate  was  considerably  relaxed,  and  par¬ 
ticularly  for  off-the-shelf  products.  Sharp  reductions  in  the  Pentagon's 
procurement  workforce  in  the  1990s  simultaneously  eroded  in-house 
technical  expertise,  albeit  with  little  contemporaneous  worry,  for  con¬ 
tractors  were  deemed  more  than  capable  of  maintaining  their  own 
technical  data.  By  the  late  1990s,  the  practice  reached  its  apex  in  the 
Total  System  Product  Responsibility  (TSPR)  concept,  in  which  a  single 
contractor  was  paid  for  the  delivery  and  long-term  maintenance  of  a 
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system,  in  a  single  long-term  contract.  The  agency  problems  in  that 
approach  led  to  some  spectacular  failures,  such  as  the  ongoing  debacle 
of  the  Space-Based  InfraRed  System  (SBIRS;  see  Hasik,  2004)— still 
unavailable  for  its  deemed  role  in  ballistic  missile  detection  and  track¬ 
ing,  some  15  years  after  its  inception. 

The  Bush  administration  almost  entirely  continued  the  Clinton 
administration's  policy,  though  it  had  largely  backed  away  from  the  TSPR 
concept  by  the  end  of  its  second  term.  Congress  was  busy  rewriting  laws 
as  well.  In  2007,  the  John  Warner  National  Defense  Authorization  Act 
mandated  that  PMs  of  MDAPs  assess  'The  merits  of  a  priced  contract 
option  for  the  future  delivery  of  technical  data  that  were  not  acquired 
upon  initial  contract  award,  and  the  potential  for  changes  in  the  sustain¬ 
ment  plan  over  the  life  cycle  of  the  system”  (Government  Accountability 
Office  [GAO],  2011,  p.  9);  note  that  Senator  Warner's  has  been  a  popular 
name  to  invoke  in  these  matters).  In  2009,  the  WSARA  had  yet  more  fully 
declared  a  new  policy,  and  the  following  year  Better  Buying  Power  had 
effectively  declared  "TSPR  RIP''  (Grundman,  2010,  p.  4).  At  that  point, 
in  policy  directives  enacted  even  before  Carter's  (2010)  memorandum 
was  written,  PMs,  according  to  GAO's  (2011)  report,  were  required  to: 

1.  assess  the  data  required  to  design,  manufacture,  and  sus¬ 
tain  the  system  as  well  as  to  support  re-competition  for 
production,  sustainment,  or  upgrade; 

2.  address  the  merits  of  including  a  priced  contract  option  for 
future  delivery  of  data  not  initially  acquired; 

3.  consider  the  contractor's  responsibility  to  verify  any  asser¬ 
tion  of  restricted  use  and  release  of  data;  and 

4.  address  the  potential  for  changes  in  the  sustainment  plan 
over  the  life  cycle  of  the  weapon  system  or  subsystem,  (p.  11) 

With  Carter's  emphasis,  PMs  would  henceforth  think  long  and  hard 
about  the  data  and  the  data  rights— if  they  could  quite  understand  the  dif¬ 
ference,  the  advantages,  and  which  benefits  might  remain  elusive. 
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Historical,  Challenging  Cases  in  the  Technical  Data 
Approach 

Buying  data  is  not  a  panacea  simply  because  intellectual  capital  (IC) 
is  not  synonymous  with  intellectual  property  (IP)  (Gallop,  2011,  p.  38).  As 
noted  earlier,  possession  of  the  data  does  not  necessarily  confer  rights  to 
the  data.  Conversely,  the  rights  to  produce  a  system  may  exist  separate 
from  the  data  needed  to  do  so.  Moreover,  neither  IC  nor  IP  constitute 
individual  skills  or  organizational  knowledge  per  se,  and  some  technolo¬ 
gies  are  quite  firm-specific.  Consequently,  technical  drawings  are  just 
the  start  of  opening  a  second  source.  As  there  have  long  been  alternative 
methods  of  second-sourcing  worth  considering,  such  as  directed  licens¬ 
ing  or  functionally  equivalent  purchases,  one  analysis  (Sellers,  1983)  of 
nearly  30  years  ago  from  the  Defense  Systems  Management  College  took 
a  dim  view  of  the  salience  of  purchasing  TDPs: 

Although  theoretically  sound,  this  method  is  perhaps  the  most 
hazardous  of  all  the  second-sourcing  methodologies.  It  is  not 
well-suited  for  use  in  highly  complex  systems  or  systems  with 
unstable  designs  or  technologies,  (p.  14) 

In  other  words,  with  most  modern  weapons. 

As  an  example,  consider  the  case  of  the  Japanese  F-2  fighter  jet  pro¬ 
gram.  In  the  early  1990s,  General  Dynamics  (predecessor  in  Fort  Worth 
to  Lockheed  Martin)  began  working  with  Mitsubishi  Heavy  Industries 
to  help  produce  a  less-expensive  domestic  supplement  to  the  Japanese 
Air  Self-Defense  Force's  F-15  jet  fighters— seemingly,  a  Japanese  analog 
to  the  F-16.  Indeed,  the  F-16  served  as  the  basis  for  the  program,  with 
diagrams,  production  licenses,  and  technology  transfer  assistance  forth¬ 
coming.  Howls  continued  for  some  time  about  the  ''giveaway  of  advanced 
aerospace  technology  to  America's  most  relentless  rival"  (Lorell,  1995a, 
p.  2),  but  the  eventual  result  was  unimpressive  (Garretty,  2002,  pp.  35-37, 
42-43).  Between  1995  and  2011,  following  the  initial  prototypes,  only  94 
combat-capable  F-2s  were  built,  in  a  60/40  work-sharing  agreement  with 
Lockheed  Martin,  for  approximately  $104  million  each.  In  short,  for  all 
its  trouble,  the  Japanese  government  got  not  a  lot  of  technology  transfer, 
and  a  shockingly  expensive  derivative  of  an  otherwise  economical  air¬ 
plane.  The  U.S.  General  Accounting  Office  concluded  as  early  as  1992  that 
the  technology  transfer  process  for  the  F-2  had  simply  been  "too  strict" 
for  cost-effective  coproduction  (Lorell,  1995b,  p.  361). 
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Consider  further  the  case  of  an  American  purchase  of  a  TDP,  gone 
very  wrong:  that  of  the  M119 105  mm  howitzer,  nee  the  British  Light  Gun. 
In  the  late  1980s,  the  U.S.  Army  bought  the  TDP  and  the  TDRs  from  Royal 
Ordnance  (RO),  the  British  government's  arsenal  for  artillery  and  muni¬ 
tions,  for  licensed  production  at  the  Watervliet  and  Rock  Island  Arsenals. 
Management  at  RO  did  not  fully  understand  what  the  U.S.  Army  meant 
by  a  TDP,  as  howitzer  production  for  the  rather  smaller  British  Army  was 
a  craft-oriented,  fix-it-on-the-shop-floor  process.  Though  RO  was  then  a 
crown  corporation  of  an  allied  state,  the  U.S.  Army  considered  suing  the 
organization  for  providing  a  package  wholly  inadequate  for  establish¬ 
ing  a  new  production  line  with  fully  trained  workers  (Schaller,  1996,  p. 
42).  The  TDP  was  also  technically  inaccurate:  its  original  estimate  for 
tooling  costs  was  $8  million,  but  actual  costs  eventually  exceeded  $23 
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million.  The  license  fee  from  RO  for  the  TDRs  was  initially  just  £1.15 
million,  but  RO's  subsequent  charge  for  fixing  the  deficiencies  in  the 
TDP— representing  scores  of  engineering  man-years  for  which  it  had 
not  originally  been  contracted— was  $4.75  million.  Accordingly,  the 
Army's  attempt  at  concurrent  engineering  at  Rock  Island  went  particu¬ 
larly  badly,  and  even  after  that,  the  Army's  Research,  Development,  and 
Engineering  Center,  Watervliet,  and  Rock  Island  spent  another  $3  mil¬ 
lion  fixing  yet  further  deficiencies  in  the  TDR 

As  noted,  original  sources  have  a  natural,  built-in  advantage  of  tacit 
knowledge  about  their  own  products,  and  whether  omissions  from  TDPs 
are  just  omissions  or  conscious  commissions  of  opportunism,  defects 
therein  are  typically  ''almost  always  the  case"  (Sellers,  1983,  p.  14;  see 
also  Witte,  2002).  Those  fighters  and  howitzers  serve  today,  but  as  the 
examples  show,  even  ultimate  successes  with  TDPs  can  cost  "an  incred¬ 
ible  amount  of  time  and  money"  (Schaller,  1996,  p.  39). 

Back  in  the  1980s,  the  Army's  project  officer  for  the  M119  was  a  junior 
civil  servant  named  Kevin  Fahey.  Today,  Fahey  is  the  Department  of  the 
Army's  program  executive  officer  (PEO)  for  Ground  Combat  Systems,  and 
an  official  pushing  to  procure  technical  data,  rights  and  all.  In  2009,  pur¬ 
suant  to  the  WSARA,  his  staff  began  calling  Army  contractors  possessing 
proprietary  designs  to  inquire  about  buying  what  it  could.  Reports  from  at 
least  one  contractor  reveal  a  remarkable  lack,  at  least  initially,  of  economic 
sense  on  the  government's  negotiating  team.  The  Army's  initial  position 
presumed  (and  apparently  innocently)  that  the  contractor  would  only 
seek  compensation  for  the  engineering  man-hours  needed  to  reproduce 
the  drawings.  When  apprised  of  the  need  to  pay  separately  for  the  rights, 
in  compensation  for  possibly  lost  future  profits,  the  Army's  negotiators  did 
quickly  come  around  (anonymous,  personal  communication,  2009). 

That  is,  if  the  government  is  investing  in  a  competitive  process,  the 
potentially  displaced  contractors  will  assign  the  avoidance  of  that  pro¬ 
cess  some  value.  That  technology  transfer  is  costly,  both  in  purchase 
price  and  learning  costs,  and  so  paybacks  on  this  investment  have  been 
observed  generally  to  take  at  least  3  years  (Daly  &  Schuttinga,  1982,  p.  63). 
Statistical  estimates  of  learning  curves  tend  to  be  highly  unreliable  (see 
Alchian,  1963),  and  can  even  turn  negative  with  "organizational  forget¬ 
ting"  (Benkard,  2000),  so  the  error  range  on  those  payback  estimates 
can  be  considerable.  The  government  must  also  maintain  that  internal 
expertise,  and  continue  to  update  the  TDP  as  the  system  is  upgraded 
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over  time  (Sellers,  1983,  p.  14).  With  problems  like  these,  unsurprisingly, 
second  sourcing  has  historically  been  used  less  to  reduce  price  than  to 
deal  with  primary  suppliers'  quality  problems  (see  Lyon,  2006). 

A  Simple  Economic  Model  for 
Pricing  Technical  Data 

Price,  though,  is  the  emphasis  of  Better  Buying  Power,  and  so  pur¬ 
chasing  data  have  become,  whether  at  Fahey  or  Carter's  direction,  a 
proactive  and  presumptive  option.  Buying  data  early  in  a  program  (as 
Carter's  [2010]  memorandum  directs)  is  very  appealing  (House  Armed 
Services  Committee,  2010,  p.  8),  but  pricing  is  problematic  for  equip¬ 
ment  already  in  production.  Much  of  the  volume  of  procurement  in 
defense,  after  all,  is  not  for  wholly  new  systems,  but  for  new  units  of 
systems  already  fielded,  or  for  modifications  to  those  systems.  With  its 
monopsony  power,  the  government  can  apply  implicit,  even  unwitting 
pressure  on  contractors  who  sell  largely  to  military  customers.  Loss  of 
goodwill  from  outright  refusal  of  a  sale  at  any  price  maybe  unpalatable. 


FIGURE  1.  TWO  CONTRACTORS,  FIXED  MARGIN,  KNOWN  COSTS 
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Facing  that  double  bind,  a  contractor  may  seek  a  defensible  position  by 
marketing  its  irreproducible  tacit  knowledge,  inextricably  proprietary 
processes,  and  efficient  embeddedness  with  the  customer  (Uzzi,  1996). 

How  the  PM  will  respond  is  more  problematic.  Buying  data  only 
makes  sense  if  the  second  run  can  amortize  the  costs  of  the  produc¬ 
tion  shift  plus  the  purchase  of  that  data.  The  quantities  and  prices  to 
which  contractors  would  respond  optimally  are  likely  less  clear.  Dixit 
(2002,  pp.  707-708)  provides  the  beginnings  of  a  model,  stipulating  a 
competition  involving  two  contractors,  with  higher  and  lower  internal 
costs.  The  government  may  have  buying  power,  but  it  does  not  know 
which  contractor  is  which,  and  so  the  lower  cost  contractor  can  repre¬ 
sent  itself  as  higher  cost,  and  conceivably  earn  fat  margins.  Auditing 
under  profit  regulations  can  drive  down  this  margin  somewhat,  but  only 
imperfectly:  management  can  pad  its  accounts  with  featherbedding,  and 
slack  off  from  the  pursuit  of  factor  efficiencies,  buying  a  comfortable 
life  at  public  expense.  Dixit's  assumptions  here  are  quite  plausible  in 
studying  military  procurement,  where  the  government  often  has  quite 
imperfect  information  of  any  contractor's  cost  or  quality.  However,  his 
prescribed  solution,  a  menu  of  two  price-and-quantity  combinations 
by  which  the  contractors  will  efficiently  self-identify  as  either  high-  or 
low-cost,  is  essentially  unknown  in  our  realm. 

And  yet,  if  we  used  his  model,  we  would  still  not  have  considered  pric¬ 
ing  the  data,  for  the  government  must  pay  in  advance  simply  to  hold  the 
competition  at  all.  To  accommodate  this  complication,  and  to  conform  to 
a  recognizable  military  procurement  mechanism,  I  offer  an  alternative 
model,  shown  in  Figure  1.  Here,  the  PM  attempts  to  procure  a  certain 
quantity  of  weapons,  specified  by  budget  planners,  minimizing  cost,  in 
a  single  round  of  procurement.  Arriving  at  Milestone  B,  the  PM  has  an 
offer  of  Q  units  at  total  cost  P  from  the  incumbent  contractor,  which  has 
designed  and  prototyped  the  weapon.  The  PM,  however,  suspects  the 
incumbent  to  be  high-cost.  A  competent  second  source,  thought  by  the 
PM  to  be  low-cost,  markets  its  capabilities  as  a  production  alternative, 
notionally  at  price  P'.  (We  assume  invariant  quality  between  contrac¬ 
tors.)  The  respective  firm-specific  cost  curves  for  the  weapon  are  shown 
with  marginal  costs  (slopes)  cHC  and  cLC.  For  simplicity,  we  assume  the 
same  fixed  start-up  cost  CF,  and  a  constant  margin,  fixed  and  audited  by 
the  government,  of  P  -  C  =  P'  -  C'.  The  lower  cost  firm  is  thus  not  incentiv- 
ized  to  pad  its  costs  for  a  greater  prize. 
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Purchasing  data  rights  could  activate  the  second  source,  which  would 
then  presumably  win  the  competition.  By  inspection,  we  see  that  the  PM's 
reservation  price  for  the  data  is  C  -  C'  (equivalent  to  P  -  P'  with  the  fixed 
margin).  We  imagine  that  the  incumbent  might  sell,  even  if  selling  means 
exit:  the  profit  margin  P  -  C  can  be  taken  as  the  incumbent's  reservation 
price.  If  the  incumbent  ascertains  in  advance  that  this  margin  will  not 
exceed  the  government's  reservation  price  (if  C  -  C'  >  P  -  C),  then  nego¬ 
tiations  are  possible.  Because  we  have  stipulated  a  fixed  margin  of  P  -  C 
=  P'  -  C',  by  substitution  the  preceding  condition  reduces  to  P'  <  C.  That 
is,  if  the  challenger's  price  is  less  than  the  incumbent's  cost,  and  profit  is 
the  contractor's  primary  motivation,  the  incumbent  and  the  government 
may  make  a  deal.  It  is  important  to  note,  though,  that  if  the  cost  curves 
are  close,  buying  out  the  incumbent's  margin  will  save  the  customer  little. 


If  the  challenger^s  price  is  less  than  the  incumbent’s 
costy  and  profit  is  the  contractor ’s  primary 
motivation,  the  incumbent  and  the  government 
may  make  a  deal. 


Tools,  Rules,  and  Schools  on  the  Path 
Towards  Technical  Data 

This  is  all  simple,  but  again  problematic,  for  neither  the  PM  nor 
the  second  source  can  be  certain  of  C',  and  thus  P'.  The  PM  presumably 
understands  the  broad  technical  nature  of  the  program,  but  the  PM  is 
not  a  production  manager,  with  the  same  operational  understanding  of 
a  PM's  contractor  counterpart.  The  PM's  should-cost  analysis  (directed 
separately  in  Better  Buying  Power)  may  help  understand  those  costs,  but 
anecdotally,  the  should-cost  analysis  does  not  begin  well.  Commercial 
best  practice  in  supplier  management  holds  that  should-cost  briefings 
should  be  transparent  in  their  assumptions  and  analyses,  so  that  sup¬ 
pliers  can  correct  customers'  misimpressions  and  find  common  ways  to 
remove  costs  from  the  shared  value  chain.  According  to  one  prominent 
analyst,  in  several  cases  the  government's  men  have  been  dropping  the 
should-cost  figure  on  the  table  and  declining  to  provide  further  insight 
(B.  Callan,  personal  communication,  2012).  This  is  no  way  to  do  business. 
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The  soliciting  second  source  also  presumably  understands  its  own 
production  capabilities  generically,  but  lacks  the  specific  TDP  and  the 
tacit  knowledge  built  during  the  development  of  the  system.  The  original 
source  might  know  its  own  production  capabilities  reasonably  well,  but 
not  precisely,  for  it  is  only  now  planning  to  bring  the  weapon  into  actual 
production.  The  original  source,  therefore,  will  presumably  know  even 
less  about  the  second  source’s  costs.  Even  in  the  best  of  times,  the  original 
source  commonly  calculated  breakpoints  of  economic  order  quantities, 
economic  production  rates,  and  minimum  sustaining  rates,  despite  their 
official  acronyms  and  emphasis  in  the  education  of  the  PM,  can  be  “surpris¬ 
ingly  difficult  to  pin  down”  (Schilling,  Hagewood,  Snodgrass,  &  Czech,  2011, 
p.  43).  If  just  the  estimated  slopes  of  the  cost  curves  differ  from  reality,  any 
of  the  players  in  the  game  may  find  themselves  in  the  situation  depicted  in 
Figure  2,  making  decisions  on  faulty  information,  with  hazardous  results. 

The  key,  to  cite  Decker-Wagner  (2011),  is  hewing  to  that  “estimated 
risk-reward”  (p.  105).  Cost-benefit  analyses  can  get  complicated  if  policy¬ 
makers  have  “highly  unstable  and  often  incomprehensible”  preferences 
(Zaharidias,  2008,  p.  517).  All  parties  may  wonder  if  the  budget  will  fall,  or 
gyrate  from  year  to  year.  The  PM  will  likely  last  in  the  job  for  but  a  few  years, 
while  the  PM’s  counterparts  may  stay  in  their  jobs  for  many  more;  if  the 
contractor’s  rationality  is  bounded,  the  PM’s  may  be  more  so.  The  problem 
is  thus  tripartite,  strategic,  probabilistic,  asymmetric,  and  frankly  quite 
challenging.  Without  a  clear  path  to  a  solution,  and  slight  punishments  for 
dodging  policies,  PMs  could  just  walk  away  from  the  problem.  Indeed,  many 
have.  In  the  GAO’s  (2011)  audit  (p.  13)  of  compliance  with  the  2007  and  2009 
statutes,  none  of  the  12  program  management  offices  sampled  had  fully 
undertaken  all  four  of  the  mandated  analyses  of  data  rights  acquisition. 

Alternatively,  if  pushed  hard,  but  without  the  analytical  tools  to 
tackle  the  question,  the  policy  of  just  considering  buying  technical  data 
could  become  inefficiently  self-executing.  Without  data  rights,  competi¬ 
tion  is  more  challenging,  but  competition  is  the  clear  dictum  of  Better 
Buying  Power.  Buying-in  reduces  organizational  uncertainty,  even  if  the 
actual  business  case  is  marginal,  which  can  appeal  more  to  the  PM,  as 
agent,  than  the  PM’s  principal— the  PEG  or  assistant  secretary.  If  buying 
data  means  “sticking  it  to  the  contractors,”  the  newly  emphasized  prac¬ 
tice  can  become,  in  Selznick’s  famous  phrase,  “infused  with  value  beyond 
the  technical  requirements  of  the  task  at  hand”  (1957,  p.  17).  PMs  might 
then  underthink  and  overreact,  refiexively  offering  to  buy  technical  data 
at  heuristically  determined  prices.  In  a  rush  to  do  a  deal,  these  prices 
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may  prove  excessive,  and  thus  sticking  it  to  no  one  but  the  government 
itself.  To  defend  the  work,  the  program  offices  may  simply  shade  their 
business  cases  to  justify  their  preferred  paths  of  less  resistance.  And 
thus,  we  would  find  defense  fulfilling  Behn's  assertion  (1995,  p.  321)  that 
''constraining  people  from  doing  anything  wrong  often  simultaneously 
constrains  them  from  doing  anything  right.'’ 

Fearing  that  the  memorandum's  guidance  could  become  ossified 
as  such  a  presumed  rule  (Buy  all  TDPs!),  the  current  Under  Secretary 
of  Defense  for  Acquisition,  Technology  and  Logistics  (USD[AT&L]) 
Frank  Kendall  has  "tried  hard  to  communicate. ..that  our  guidance  is 
just  that— guidance"  (Kendall,  2011a,  p.  3).  If  Better  Buying  Power  aims 
to  implement  23  game -changing  elements  of  guidance,  then  in  the  words 
of  one  reporter  at  the  roll-out  press  briefing  (Gates  &  Carter,  2010),  this 
"might  seem  to  require  a  cultural  change  within  the  Department  of 
Defense."  The  response  by  the  former  USD(AT&L)  was  noncommittal, 
with  the  emphasis  on  changing  behavior  versus  cultural  change: 


FIGURE  2.  TWO  CONTRACTORS,  FIXED  MARGIN,  UNKNOWN 
COSTS 


QUANTITY 
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DR.  CARTER  Cultural  change  is— I  always  say  I  don't  do  cultural 
change;  it's  too  hard.  So  we're— this  is  directing  specific  actions. 
And  the  actions  that  we  want  are  pretty  specific,  and  the  cause- 
and-effect  is  pretty  specific,  I  think  you'll  find  as  you  read  this, 
and  the  metrics  by  which  we  measure  the  effects  are  spelled  out 
in  the  document.  So  culture's  too  hard  for  me.  Behavior— that's 
what  we're  after,  (p.  3) 


Conclusions  and  Recommendations 

Although  Carter's  (2010)  memorandum  lays  out  metrics  for  other  ini¬ 
tiatives,  it  does  no  such  thing  regarding  data  rights— one  of  a  PM's  more 
challenging  economic  analyses.  If  there  is  direction,  it  is  toward  mere 
consideration.  To  be  sure,  issuing  new  formal  rules  could  be  counterpro¬ 
ductive,  as  the  PM's  managerial  judgment  under  uncertainty  is  essential 
to  the  pursuit  of  better  value.  Given  the  complexity  of  the  business  case 
that  the  USD(AT&L)  now  demands  at  every  MDAP's  Milestone  B,  some 
better  tools  would  be  important. 

Better  still  might  be  actually  tackling  what  Carter  calls  “too  hard"— 
the  culture.  While  complex  weapons  acquisition  should  be  an  eyes-open 
process,  it  simply  cannot  be  an  arms-length  transaction.  The  Pentagon's 
procurement  institutions  should  inculcate  in  managers  a  strategic  sense 
for  mutually  dependent  relationships  with  long-term  incentives  for  sus¬ 
tained  innovation.  The  pursuit  of  such  fuzzy  objectives  by  PMs  best  relies 
on  informal  rules  subject  to  the  judgment  of  more  senior  officials  (see 
Ingraham,  Moynihan,  &  Andrews,  2008),  and  the  conscious  development 
of  an  organizational  culture  congruent  with  the  leadership's  objectives 
in  a  commonly  understood  “unity  of  purpose”  (Gulick,  1937,  p.  39). 

Change  would  require  adjusting  the  government's  ways  of  think¬ 
ing— its  fundamental  school  of  thought  about  supplier  management. 
Carter  effectively  introduced  a  slew  of  new  rules  into  the  Pentagon's 
bureaucracy,  but  he  and  his  successor  have  developed  few  mechanisms 
for  affecting  the  behavioral  change  beyond  issuing  a  memorandum. 
Exhortations  are  no  way  to  develop  a  sound  business  process  (Deming, 
1982,  pp.  65-70),  much  less  to  develop  23  such  processes.  Pricing  techni¬ 
cal  data  in  each  of  the  Pentagon's  programs  is  but  one  of  those,  and  a  very 
challenging  one  at  that. 
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Initial  Capabilities  Documents: 
A  10-Year  Retrospective  of 
Tools,  Methodologies,  and  Best 

Practices 
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The  Joint  Capabilities  Integration  &  Development  System 
(JCIDS)  is  10  years  old  and  ripe  for  reviev\/.  A  central  output 
document  of  the  JCIDS  process  is  an  Initial  Capabilities 
Document  (ICD)  used  by  the  Department  of  Defense  to 
define  gaps  in  a  functional  capability  area  and  define 
ne\A/  capabilities  required.  The  research  team  analyzed  10 
years  of  ICDs  to  identify  methods  and  trends.  The  team 
found  that  several  methodologies  were  favored  and  a 
convergence  emerged  in  format  and  necessary  content. 
Additionally,  potential  shortfalls  in  current  best  practices 
of  interest  to  implementors  and  decision  makers  are 
identified.  Guidelines  and  best  practices  are  presented 
to  create  more  effective,  concise,  and  complete  ICDs. 
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As  a  historically  unstable  process,  it  may  come  as  a  surprise  to 
many  acquisition  practitioners  that  the  formal  written  procedures 
and  processes  that  embody  the  Defense  Acquisition  System  and  Joint 
Capabilities  Integration  and  Development  System  (JCIDS)  are  now  over 
10  years  old.  During  this  time,  the  Department  of  Defense  (DoD)  has 
published  significant  revisions  and  updates  to  the  JCIDS-related  docu¬ 
ments,  including  Department  of  Defense  Instruction  (DoDI)  5000.02 
and  the  Joint  Capabilities  Integration  8c  Development  System  Manual 
(DoD,  2013;  Joint  Requirements  Oversight  Council  [JROC],  2012).  The 
current  system's  longevity  may  be  partially  attributable  to  its  utiliza¬ 
tion  of  modern  management  approaches,  further  enabled  by  a  slow 
convergence  of  the  Joint  Strategic  Planning  System  set  in  motion  by 
the  Goldwater-Nichols  Act  (Goldwater-Nichols,  1986).  With  its  focus 
on  joint  development  and  deconfiiction  of  capabilities,  JCIDS  uses  a 
portfolio  management  approach  and  streamlined  documentation  to 
elevate  user  requirements  relatively  quickly  and  vet  them  against  cur¬ 
rent  capabilities.  Further,  its  emphasis  on  knowledge  management 
ensures  that  all  stakeholders  can  view  the  process  and  its  outcomes  as 
the  key  documents  percolate  through  the  JCIDS  process. 

Early  analysis  of  the  JCIDS  process  by  the  U.S.  Government 
Accountability  Office  (GAO,  2008)  identified  variable  product  quality. 
Attempts  were  made  at  creating  user's  guides  to  improve  document 
quality  (JROC,  2012;  Joint  Chiefs  of  Staff  [JCS],  2009);  however,  these 
documents  did  not  fully  address  the  analysis  techniques  contained 
therein.  As  a  key  component  of  process  quality,  the  ability  to  select,  use, 
and  report  an  appropriate  analysis  technique  is  an  item  of  interest  for 
authors,  stakeholders,  and  portfolio  managers.  Therefore,  this  effort 
reviewed  the  content,  tools,  and  methodologies  recorded  in  the  past 
10  years'  Initial  Capabilities  Documents  (ICD)  created  as  a  part  of  the 
JCIDS  process. 


Early  analysis  of  the  JCIDS  process  by  the  U.S. 
Government  Accountability  Office  (GAO,  2008) 
identified  variable  product  quality. 
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As  one  of  the  first  products  created  in  JCIDS,  ICDs  are  important 
because  they  validate  requirements  derived  through  an  analysis  of 
current  capabilities  and  capability  gaps.  Additionally,  they  are  signed 
by  senior  service  members  and  are  the  basis  for  program  acquisitions. 
Further,  due  to  their  recommended  brevity,  it  is  important  that  ICDs 
contain  the  correct  level  of  detail  to  identify  the  key  assumptions, 
limitations,  and  boundary  conditions  contained  or  referenced  in  their 
analyses.  A  lack  of  analytical  clarity  at  this  stage  may  lead  to  misdirected 
resources  further  in  the  process  (GAO,  2008). 

Of  particular  interest  were  the  methodologies  that  implementers 
and  decision  makers  were  choosing  to  use  in  developing  ICDs.  Through 
this  process,  it  was  possible  to  identify  a  series  of  best  practices  and 
guidelines  to  improve  ICD  quality,  and  thus  aid  in  the  evolution  of  JCIDS. 

Background 

The  JCIDS  process  was  created  as  a  response  to  a  2002  memo¬ 
randum  from  the  Secretary  of  Defense  to  the  Vice  Chairmen  of  the 
Joint  Chiefs  of  Staff  to  study  alternative  ways  to  evaluate  requirements 
(JCIDS,  2014).  At  the  time  of  this  memorandum,  the  governing  document 
was  Chairman  of  the  Joint  Chiefs  of  Staff  Instruction  3170.01B  (C  JCSI, 
2001)  and  was  titled  the  Requirements  Generation  System.  The  purpose 
of  JCIDS  was  to  streamline  and  standardize  the  methodology  to  identify 
and  describe  capabilities'  gaps  across  the  DoD,  and  to  engage  the  acqui¬ 
sition  community  early  in  the  process  while  improving  coordination 
between  departments  and  agencies. 

The  GAO's  (2008)  report  indicated  that  “the  JCIDS  process  has  not 
yet  been  effective  in  identifying  and  prioritizing  warfighting  needs  from 
a  joint,  department-wide  perspective”  (GAO,  2008,  para.  1).  This  report 
outlined  the  shortfalls  and  gaps  in  the  JCIDS  process  in  its  5-year  life 
span,  furthering  the  redesign  of  the  process.  Additionally,  the  report 
outlined  several  recommendations  for  the  DoD,  including  developing 
a  more  analytical  approach  within  JCIDS  to  better  prioritize  and  bal¬ 
ance  capability  needs  as  well  as  allocating  the  appropriate  resources  for 
capabilities  development  planning. 

The  current  documentation  for  both  creating  and  implementing  ICDs 
are  the  Capabilities-based  Assessment  (CBA)  User's  Guide  and  the  JCIDS 
manual.  These  documents  were  released  in  2009  and  2012  respectively 
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as  a  part  of  the  process  to  address  the  issues  found  by  the  2008  GAO 
report.  The  impact  of  these  documents  in  improvements  to  the  JCIDS 
process  has  yet  to  be  determined,  but  will  be  discussed  in  this  article. 

Focus  and  Methodology 

The  research  team  used  the  Knowledge  Management/Decision 
Support  (KM/DS)  system  to  examine  the  JCIDS  process.  The  KM/DS 
Web  site  is  the  repository  for  the  documents  created  through  or  as  a 
byproduct  of  the  JCIDS  process.  Included  in  this  study  are  ICDs,  Joint 
Capabilities  Documents  (JCD),  Capability  Development  Documents, 
and  other  supporting  documents  that  are  a  part  of  this  process.  To  focus 
this  research,  the  team  specifically  studied  the  core  documents— ICDs 
and  JCDs— to  better  understand  what  kinds  of  methodologies  are  being 
implemented  by  the  various  Services  to  convey  the  gap  information 
under  study. 

Cf  those  entered  in  the  KM/DS  system,  there  were  over  1,000  ICDs 
and  JCDs  in  various  phases  of  the  JCIDS  process  covering  the  period 
January  1,  2002  to  December  31,  2012.  The  team  decided  to  focus  on 
only  those  documents  that  were  considered  'Validated'  and  'Final,'  with 
the  expectation  of  little  to  no  revision  remaining  for  these  documents 
in  the  near  future.  These  criteria  reduced  the  number  of  the  documents 
under  review  to  225  ICDs/JCDs.  The  team  of  four  researchers  split  the 
ICDs/JCDs  evenly  across  year  and  type  to  ensure  similar  exposure  to 
the  complete  population  available.  At  the  completion  of  the  review,  the 
researchers  met  and  discussed  commonalities  and  anomalies  found  in 


TABLE  1.  ATTRIBUTES  FOR  ANALYSIS 
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documents  of  interest,  and  in  the  population  in  general.  For  purposes 
of  this  article,  the  term  ICD  will  be  used  to  describe  both  the  ICDs  and 
JCDs  unless  specified  otherwise. 

The  team  formulated  an  initial  set  of  generally  accepted  methodologies 
for  a  baseline  to  identify,  categorize,  and  sort  the  currently  used  methodolo¬ 
gies  within  the  ICDs.  They  did  not  solely  consider  this  set  of  techniques,  but 
allowed  for  an  expansion  of  the  list  to  detect  emergent  techniques. 


Ultimately,  it  was  the  intention  of  the  research 
team  to  observe  and  report  on  best  practices  for 
future  ICD  writers. 


Additionally,  an  analysis  was  performed  on  key  metrics  and  areas  of 
interest  to  see  if  there  were  any  correlations  or  observations  that  could  be 
made  about  various  components  of  the  ICDs.  These  attributes  were  cho¬ 
sen  as  they  were  key  areas  of  interest  or  sections  in  the  Capabilities-based 
Assessment  (CBA)  User's  Guide  and  the  JCIDS  manual.  By  examining 
these  attributes,  the  team  was  able  to  determine  to  what  extent  past  ICDs 
have  followed  current  guidance.  Some  of  the  components  considered  in 
the  analysis  can  be  found  in  Table  1. 

Ultimately,  it  was  the  intention  of  the  research  team  to  observe  and 
report  on  best  practices  for  future  ICD  writers.  As  such,  we  focused 
on  finding  those  ICDs  that  best  embodied  the  intentions  found  in  the 
Capabilities-based  Assessment  (CBA)  User's  Guide  (JCS,  2009)  and  the 
JCIDS  manual  (JROC,  2012). 


Results 

The  team  examined  several  ICD  characteristics  that  are  presented 
in  the  JCIDS  manual  and  were  expected  to  be  used  in  most  ICDs  (Figure 
1).  The  team  found  that  of  the  features  prescribed  by  the  JCIDS  manual, 
many  were  not  present  in  the  majority  of  ICDs  reviewed.  Less  than  half 
of  the  ICDs  described  what  analysis  was  done  to  identify  capability  gaps. 
Over  90  percent  of  the  ICDs  reviewed  define  a  specific  capability  while 
some  ICDs  do  not  have  a  well-defined  end  state. 
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Nearly  half  of  the  ICDs  analyzed  defined  their  Measures  of 
Effectiveness  (MOE),  described  their  analysis,  prioritized  gaps  and  capa¬ 
bilities,  and  defined  minimum  values  for  required  capability  attributes. 
The  presence  of  these  characteristics  provides  additional  information 
to  the  reader  and  improves  the  fidelity  of  the  ICD;  their  absence  leaves 
commonly  questioned  areas  open  for  discussion.  The  2012  JCIDS  man¬ 
ual  requires  threshold  values,  but  description  of  the  analysis  has  been 
left  open  to  the  document  creator,  and  many  choose  not  to  describe  it. 
In  fact,  the  manual  states  a  preference  to  ”avoid  unnecessary  rigor  and 
time-consuming  detail.”  Applying  and  documenting  some  level  of  rigor 
seems  necessary  and  useful  for  documenting  how  gaps  were  identified 
and  showing  how  the  capability  requirements  were  justified.  The  priori¬ 
tization  of  gaps  and  capabilities  helps  decision  makers  understand  those 
components  that  are  critical  when  resources  are  limited  to  address  the 
full  capability  gap,  but  allows  for  partial  capability  fulfillment  or  a  subset 
of  smaller  gaps  to  be  filled. 

The  inclusion  of  an  Analysis  of  Alternatives  (AoA)  is  an  interesting 
additional  piece  of  content  as  it  is  no  longer  part  of  the  Capabilities-based 
Assessment  (CBA)  User's  Guide,  and  is  done  in  subsequent  work  of  the 


FIGURE  1.  ICD  CONTENT  ANALYSIS 


Note.  AoA  =  Analysis  of  Alternatives;  MOEs  =  Measures  of  Effectiveness;  UJTL  = 
Universal  Joint  Task  List. 
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JCIDS  process.  Nearly  one-third  of  all  ICDs  included  some  form  of  an 
AoA,  whether  in  the  form  of  a  brief  paragraph  or  a  full  documentation 
found  in  attachments  or  enclosures.  Most  documents  that  contained  a 
complete  AoA  were  from  the  first  5  years,  a  period  of  time  in  which  the 
content  of  ICDs  was  still  in  fiux.  Including  an  AoA  would  presuppose  a 
preferred  materiel  solution— something  not  within  the  scope  of  docu¬ 
menting  a  capability  gap. 

Also,  less  than  25  percent  of  the  ICDs  surveyed  contained  objective 
values  for  the  capabilities  to  be  met.  While  it  has  become  more  common 
for  threshold  values  to  be  defined  for  capabilities,  objective  values  can 
only  be  seen  in  less  than  half  of  those  cases.  One  might  expect  to  see 
objective  values  used  more  frequently  to  quantify  desired  capabilities 
beyond  the  minimums.  Including  objective  values  is  expected  to  aid  the 
process  owner  in  determining  if  a  recommended  solution  is  able  to  meet 
the  objective  of  closing  the  specified  gap. 

FIGURE  2.  NUMBER  OF  FUNCTIONAL  CAPABILITIES  IN  ICDS 

ANALYZED 


Functional  Capabilities  Board 


■  SUPPORTING 

■  LEAD 

«  LEAD  CUMULATIVE 
—  SUPPORTING  CUMULATIVE 
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Identifying  the  Functional  Capabilities  Boards  (FCB)  to  which 
ICDs  were  assigned  provided  insight  as  to  what  types  of  capabili¬ 
ties  have  been  defined  and  what  priorities  have  been  dictated.  FCB 
and  associated  Joint  Capability  Area  (JCA)  categories  include 
Force  Support  (formerly  Force  Support  and  Building  Partnerships); 
Battlespace  Awareness;  Force  Application;  Logistics;  Command, 
Control,  Communications,  and  Computers  (C4)/Cyber  (formerly 
Net-Centric,  Command  and  Control,  and  C4/Cyber);  and  Protection. 
Previous  FCBs,  including  Special  Operations  and  Test,  are  listed  in 
Figure  2  under  ''Other  Legacy  FCBs.” 


Identifying  the  Functional  Capabilities  Boards 
(FCB)  to  which  ICDs  were  assigned  provided  insight 
as  to  what  types  of  capabilities  have  been  defined 
and  what  priorities  have  been  dictated. 


FIGURE  3.  AVERAGE  NUMBER  OF  PAGES  FOR  ICDS  IN 
CORRESPONDING  YEARS 


Average  Pages  by  Year 


2003  2004  2005  2006  2007  2008  2009  2010  2011  2012 


AVG  PGS  =  27 
STD  DEV  =  7.58 
MEDIAN  =  22.39 
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Each  ICD  is  assigned  a  lead  and  supporting  FCB.  Figure  2  shows 
ICDs  arranged  by  lead  FCB  with  Force  Application  being  the  most  promi¬ 
nent  lead  FCB.  The  prominence  of  Force  Application  over  Force  Support 
led  the  team  to  conclude  that  validated  ICDs  are  more  likely  to  focus  on 
the  direct  needs  of  the  warfighter  and  less  likely  to  focus  on  capabilities 
of  supporting  processes.  At  the  same  time,  a  significant  number  of  ICDs 
listed  net-centricity  and  C4/Cyber  as  supporting  FCBs. 

The  research  team  decided  early  on  to  capture  the  length  of  ICDs  as  the 
Capabilities-based  Assessment  (CBA)  User's  Guide  specifically  states  that 
ICDs  should  be  no  longer  than  10  pages,  with  separate  allowance  for  appen¬ 
dices  (JCS,  2009).  Figure  3  presents  the  average  ICD  page  length  without 
appendices;  quality  and  meticulousness  were  not  necessarily  correlated 
with  quantity  of  pages.  ICDs  were  meant  to  be  concise  documents  that  out¬ 
line  the  necessary  capabilities  while  still  answering  the  required  content. 

The  drastic  increase  in  length  of  ICDs  is  potentially  a  result  of  a 
change  in  the  process  by  which  capability  gaps  were  outlined.  As  with 
most  processes,  uncertainty  in  a  new  method  allows  for  an  increase  in 
the  breadth  and  depth  of  the  information  found  within  ICDs.  As  page 
length  has  been  steadily  decreasing  over  the  last  few  years,  it  would  sug¬ 
gest  that  sponsors  have  become  more  comfortable  with  the  process  and 
have  become  more  efficient  at  outlining  the  information  needed. 

One  final  note  concerning  page  length  was  to  evaluate  the  relation  of 
page  length  to  Acquisition  Category  (ACAT)  level.  Would  larger  projects 
lend  themselves  to  taking  more  pages  to  explain  the  research  and  identify 
the  gaps?  These  two  factors  were  examined,  and  between  ACAT  Levels  I,  II, 
and  III  the  mean  page  length  was  25.53, 23.35,  and  21.02  respectively.  While 
the  difference  between  ACATs  I  and  III  are  statistically  significant  using  a 
t-test  with  an  alpha  of  .05,  the  difference  (on  average)  is  roughly  four  pages. 

Within  the  time  period  analyzed,  a  total  of  2,779  gaps  were  identi¬ 
fied;  the  average  number  of  gaps  identified  in  an  ICD  are  shown  in  Figure 
4.  Additionally,  Figure  4  illustrates  the  fiuctuation  in  the  number  of 
ICDs  validated  each  year.  The  GAO  (2008)  report  noted  that  JCIDS  was 
ineffective  in  properly  prioritizing  capabilities  and  suggested  that  nearly 
all  ICDs  submitted  were  accepted.  Since  the  inception  of  the  JCIDS  pro¬ 
cess,  2012  was  the  first  year  that  the  average  number  of  gaps  exceeded 
number  of  ICDs  validated.  This  suggests  that  ICDs  are  identifying  more 
gaps  per  document,  creating  documents  that  are  tackling  larger  and 


Defense ARJ,  July  2014,  Vol  21  No.  3 : 716-748 


724 


A  Publication  of  the  Defense  Acquisition  University 


http :  //w  w  w.  dau.  mil 


more  complex  problems  than  before.  It  appears  that  the  JCIDs  process 
has  matured,  and  the  process  has  become  more  efficient  as  a  result  of 
the  GAO  report. 

The  research  team  noted  that  many  ICDs  had  'Too  few”  gaps  identified 
(only  one  or  two,  or  none  at  all)  leading  to  the  conclusion  that  the  method¬ 
ology  employed  was  not  optimal  as  there  are  probably  more  gaps  that  have 
yet  to  be  identified,  and  several  documents  identified  "too  many”  gaps.  It 
was  very  difficult  to  understand  and  prioritize  identified  gaps  when  too 
many  were  identified  (several  contained  over  50  gaps). 

Figure  5  is  a  representation  of  the  most  frequently  used  methodolo¬ 
gies  from  2002  to  2012,  displaying  the  percentage  of  ICDs  covered  by 
the  methodology.  The  top  five  methodologies  were  chosen  for  represen¬ 
tation  as  they  represented  those  methodologies  that  were  implemented 
in  greater  than  10  percent  of  ICDs,  whereas  the  remaining  methodolo¬ 
gies  were  typically  used  in  one  to  two  ICDs  only.  Each  ICD  employed 

FIGURE  4.  AVERAGE  NUMBER  OF  GAPS  IDENTIFIED  COMPARED 

TO  NUMBER  OF  VALIDATED  DOCUMENTS 
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2003  2004  2005  2006  2007  2008  2009  2010  2011  2012 


■  AVG  NUMBER  GAPS 
—  NUMBER  VALIDATED  DOCUMENTS 


725 


Defense ARJ,  July  2014,  Vol  21  No.  3: 716-748 


Initial  Capabilities  Documents:  A  10-Year  Retrospective  of  Tools,  Methodologies,  and  Best  Practices 


several  methodologies  so  the  percentages  will  not  sum  to  100  percent. 
A  variety  of  analytical  techniques  maybe  appropriate  depending  on  the 
type  of  analysis  being  conducted.  As  an  example,  intelligence-based 
assessment  would  likely  be  an  appropriate  technique  for  identifying  a 
strategic  capability  gap  requiring  a  new  weapon  system,  but  not  appro¬ 
priate  for  identifying  the  need  for  a  new  inventory  system  for  the  Defense 
Commissary  Agency. 

Doctrine,  Organization,  Training,  Materiei,  Leadership, 
Personnei,  Faciiities-Poiicy 

The  research  team  observed  at  least  two  interpretations  of  the 
Doctrine,  Organization,  Training,  Materiel,  Leadership,  Personnel, 
Facilities-Policy  (DOTMLPF-P)  analysis  within  the  ICDs.  The  analy¬ 
sis  sometimes  took  the  course  where  ICDs  identified  DOTMLPF-P 
categories  of  nonmateriel  solutions  that  could  satisfy  capability  gaps, 
while  others  took  the  second  interpretation  where  ICDs  considered 
the  DOTMLPF-P  implications  of  their  proposed  materiel  solution. 
Defense  Acquisition  University  training  for  DOTMLPF-P  distinguishes 
between  these  uses  and  indicates  that  the  ICD  should  focus  on  the  for¬ 
mer  approach  as  the  latter  is  addressed  in  later  stages  of  the  acquisition 
process  (Defense  Acquisition  University,  2014). 


FIGURE  5.  TOP  METHODOLOGIES  USED 
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We  also  observed  a  wide  range  of  quality  in  these  analyses.  Many 
ICDs  contained  rote  statements  declaring  the  insufficiency  of  these  non¬ 
materiel  approaches  to  close  capability  gaps.  To  paraphrase  an  example, 
several  ICDs  stated  that  “DOTMLPF  solutions  were  considered...,  but 
adjustments  or  improvements  in  these  areas  will  have  minimal  impact 
to  mission  satisfaction.”  Though  not  every  capability  gap  can  be  met  with 
nonmateriel  solutions,  such  “box  check”  DOTMLPF-P  analyses  offer  no 
value  to  the  requirements  validation  process. 

In  contrast,  several  analyses  reflected  a  concerted  effort  to  find 
nonmateriel  solutions  to  supplement  the  proposed  materiel  solution. 
One  example  of  this  level  of  analysis  is  the  Air  Force's  Advanced  Pilot 
Training  ICD.  In  its  DOTMLPF-P  analysis,  the  Service  employed  a  three- 
phase  process:  first,  brainstorming  and  combining  possible  solutions; 
second,  conducting  quantitative  analysis  on  a  subset  of  the  best  of  the 
proposed  solutions;  and  third,  conducting  a  qualitative  assessment  of  the 
final  list  of  proposed  solutions.  Not  all  of  the  nonmateriel  solutions  were 
deemed  feasible  or  prudent,  but  several  were  included  as  part  of  the  final 
recommendations.  Further  explanations  of  how  the  Air  Force  conducted 
this  analysis  are  found  in  the  ICD  and  its  attachments  on  KM/DS. 

Recommendations  and  Guidelines 

Through  the  analysis  the  team  observed  a  variety  of  interpretations 
of  how  to  write  an  ICD.  In  general,  analytical  rigor  could  be  stronger. 
In  a  fiscally  constrained  environment,  the  importance  of  documenting 
analysis  is  magnified,  and  many  ICDs  fell  short  of  careful  documentation 
of  analysis.  Another  observation  is  that  most  of  the  ICDs  were  submitted 
by  the  Services  and  very  few  by  joint  sponsors.  This  is  not  surprising  as 
individual  Services  organize,  train,  and  equip  their  forces;  it  is  expected 
that  capability  gaps  will  continue  to  be  identified  by  the  Services. 

Useful  Analytical  Techniques 

Several  ICDs  utilized  subject  matter  experts  (SME)  to  identify  capa¬ 
bility  gaps  and  recommend  solutions.  One  way  to  incorporate  SME  input 
into  a  more  rigorous  fashion  is  by  employing  the  Delphi  Technique.  In 
this  method,  the  researcher  works  with  10-15  experts  to  identify,  further 
define,  and  determine  the  importance  of  an  issue  in  their  area  of  exper¬ 
tise  (Linstone  &  Turoff,  1975).  Using  the  Delphi  method  when  SMEs  are 
available  is  one  way  to  add  analytical  rigor  to  the  ICD  process. 
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Though  not  possible  for  all  ICDs,  several  documents  included  a  life- 
cycle  cost  summary  that  was  effective  in  communicating  the  costs  of 
the  capability  gap.  If  the  proposed  solution  is  expected  to  reduce  some 
recurring  cost,  presenting  those  numbers  can  make  a  convincing  case 
to  the  reader. 

In  the  Appendix  to  this  article,  the  authors  provide  a  list  of  addi¬ 
tional  analytical  techniques  along  with  a  short  description  of  each.  This 
resource  is  intended  to  assist  ICD  writers  and  project  managers  in  select¬ 
ing  a  methodology  or  methodologies  appropriate  for  their  document  or 
project.  References  are  provided  to  direct  interested  readers  to  source 
documents  with  additional  descriptions  of  each  methodology. 


One  way  to  incorporate  SME  input  into  a  more 
rigorous  fashion  is  by  employing  the  Delphi 
Technique.  In  this  method^  the  researcher  works 
with  10-15  experts  to  identify ,  further  define^  and 
determine  the  importance  of  an  issue  in  their  area 
of  expertise  (Linstone  &  Turoff^  1975). 


Architectural  Enhancements 

Nearly  all  existing  ICDs  present  a  High-Level  Operational  Concept 
Graphic  (OV-1)  depicting  the  proposed  solution(s).  A  previous  Air 
Force  Institute  of  Technology  researcher  identified  several  additional 
Department  of  Defense  Architecture  Framework  (DoDAF)  prod¬ 
ucts  that  could  be  useful  to  present  within  the  ICD  (Hughes,  2010). 
The  Capability  Taxonomy  (CV-2),  Capability  Dependencies  (CV-4), 
Capability  to  Operational  Activities  Mapping  (CV-6),  as  well  as  the 
Operational  Resource  Flow  Description  (OV-2)  and  Operational  Activity 
Decomposition  Tree  (OV-5a)  are  products  now  required  by  JCIDS  for 
the  ICD. 

Hughes  also  found  value  in  including  the  Operational  Activity  Model 
(OV-5b)  and  Operational  Activity  to  Systems  Function  (SV-5).  The  OV-5b 
presents  capabilities  and  activities  and  their  relationship  among  activities. 
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inputs,  and  outputs.  The  SV-5  maps  systems  back  to  capabilities  or  opera¬ 
tional  activities.  Neither  is  currently  recommended  in  the  JCIDS  Manual, 
but  could  be  presented  there  as  optional  architecture  products. 

Characteristics  of  Modei  iCDs 

Based  upon  analysis  of  the  data  that  were  examined  during  the  study, 
several  guidelines  or  best  practices  emerged.  The  best  written  ICDs 
provided  detailed,  but  relevant  analysis  without  being  too  wordy.  Here, 
we  propose  the  contents  of  a  model  ICD. 

The  most  fundamental  building  block  of  an  ICD  is  conformance  to 
JCIDS  standards  of  format  and  content.  The  JCIDS  Manual  presents 
a  logical  flow  of  the  document  from  gap  identiflcation  to  flnal  recom¬ 
mendations.  The  Concept  of  Operations  should  illustrate  how  the 
described  capability  will  support  the  Joint  Force  Commander.  The 
JCAs  or  Universal  Joint  Task  List  pedigree  should  be  clear,  but  not  overly 
detailed.  Documents  that  rolled  up  capability  gaps  to  Tier  2  or  Level  2 
components  seemed  more  readable  than  those  that  traced  capabilities 
to  lower  levels.  A  document  that  acknowledges  extant  systems  is  more 
convincing  in  establishing  a  capability  gap. 

The  team  believes  that  a  concise  ICD  may  be  written  with  5-12  gaps 
identifled.  Page  lengths  may  vary  by  ACAT  level,  with  more  complex 
proposed  solutions  demanding  more  explanation,  but  the  ideal  ICD 
would  be  15-25  pages  in  length.  In  short,  a  well-written  ICD  will  follow 
the  prescribed  format,  clearly  deflne  its  necessity  to  the  joint  mission, 
and  be  presented  in  a  clear  and  logical  manner.  Additionally,  the  ICD 
should  present  clear  MOEs  with  minimum  and  desired  values.  Good 
MOEs  allow  the  reader  or  evaluator  to  know  when  the  new  capability 
has  delivered  on  its  design  promises.  MOEs  are  sometimes  confused 
with  measures  of  performance  (MOP).  Noel  Sproles  states,  "MOEs 
are  concerned  with  the  emergent  properties  or  outcomes  of  a  solution. 
They  take  an  external  view  of  a  solution  and  as  such  are  different  from 
MOPs,  which  are  concerned  with  the  internal  workings  of  a  solution” 
(Sproles,  2002). 

Table  2  compares  ICD  content  required  by  the  Capabilities-based 
Assessment  (CBA)  User's  Guide,  the  JCIDS  manual,  and  recommenda¬ 
tions  based  on  our  analysis.  As  part  of  the  analysis,  the  team  identifled 
those  ICDs  that  implemented  and  followed  the  best  practices  identifled 
by  the  team.  These  ICDs,  shown  in  Table  3,  are  identifled  in  order  to 
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give  future  ICD  writers  and  functional  groups  examples  of  what  they 
can  strive  toward  to  make  clear  and  concise  documents  that  are  both 
effective  and  efficient. 

Future  Research  and  Conclusions 

Future  research  could  focus  on  the  relationship  between  the  ICD 
and  the  program  it  generates.  Can  the  utility  or  performance  of  a 
program  be  traced  to  the  description  of  the  initial  capability  gap  and 


TABLE  2.  COMPARISON  OF  CBA/ICD  CONTENT 


CBA  User's  Guide 

JCIDS  Manual 

Research  Team 

Purpose 

CONOPS/Desired 

Outcomes 

CONORS 

Background/Guidance 

Joint  Functional  Areas 

Relationship  to  Tier  2 
JCA/UJTL 

Objectives 

Description  of 

Required  Capability 
Gaps,  Overlaps, 

Redundancies 

Analysis  Techniques 
Used  with  Description 
of  Scope 

Scope 

Capability  Attributes/ 
Metrics 

Prioritized  List  of  5-12 

Capability  Gaps 

Methodology 

-Approaches 

-MOEs 

-Technological/Policy 

Opportunities 

Relevant  Threats/ 
Operational 

Environment 

Clearly  Defined  MOEs 

with  Threshold  and 

Objective  Values 

Organization/ 

Governance 

Proposals  for  Non¬ 
materiel  Solutions 

DOTMLPF-P  Analysis 

of  Non-Materiel 

Solutions 

Projected  Schedule 

Final 

Recommendations 

Clear  Final 

Recommendations 

Responsibilities 

Note.  CBA  =  Capabilities-Based  Assessment;  CONORS  =  Concept  of  Operations; 
DOTMLPF-P  =  Doctrine,  Organization,  Training,  Materiel,  Leadership,  Personnel, 
Facilities  -  Policy;  ICD  =  Initial  Capabilities  Document;  JCA  =  Joint  Capabilities 
Assessment;  JCIDS  =  Joint  Capabilities  Integration  and  Development  System;  MOEs  = 
Measures  of  Effectiveness;  UJTL  =  Universal  Joint  Task  List. 
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TABLE  3.  SAMPLE  OF  EXEMPLARY  ICDS 


Document  Name 
(Control  Number) 

Year 

Noteworthy  Items 

Data  Masked  (05- 
51947485-00) 

2005 

Layered  analytical  methods  resulted 

in  100  shortfalls  that  were  further 

clustered  and  examined-top  3 
presented  for  further  study 

Military  Operational 
Medicine  (07- 
65416952-00) 

2007 

Extensive  Doctrine,  Organization, 
Training,  Materiel,  Leadership, 

Personnel,  Facilities  (DOTMLPF);  lots 
of  prioritized  tables 

Aviation  Ground 

support  (07- 
600735309-00) 

2007 

Prioritized  tables,  quantitative 
threshold  values,  good  DOTMLPF, 
multiple  methods  used  to  determine/ 
rank  Nonmateriel  Solutions 

Initial  Capabilities 

Document  for  Joint 

Improvised  Explosive 
Device  Defeat  (07- 
66686002-00) 

2007 

Performed  a  well-documented, 
thoughtful  DOTMLPF  analysis; 

references  three  assessments— Joint 

Staff  (J8),  Joint  Improvised  Explosive 
Device  Defeat  Task  Force  baseline,  and 
follow-on;  prioritized  tables 

Biometrics  in 

Support  of  Identity 
Management  (09- 
090146111-00) 

2008 

Detailed  analysis  including  Scenario- 
Based  Planning  and  Risk  Analysis 

Advanced  Pilot 

Training  (10- 
99164267-00) 

2009 

Strong  DOTMLPF  Analysis;  clear 
explanation  of  analytical  approach 
included  in  Appendices 

Vessel  To  Shore 

Bridging  (09- 
97169105-00) 

2009 

Gaps  have  numerous  subparts;  uses  a 
typical  but  good  example  of  capability 
prioritization/mapping  matrix 
(includes  Measures  of  Effectiveness 
[MOE]  and  Minimum  values) 
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TABLE  3.  SAMPLE  OF  EXEMPLARY  ICDS  (CONTINUED) 


Document  Name 
(Control  Number) 

Year 

Noteworthy  Items 

Cross  Domain 

Enterprise  (10- 
112959174-00) 

2010 

Uses  a  typical  but  good  example  of 
capability  prioritization/mapping 
matrix  (includes  MOEs  and  Minimum 
values);  recommends  mix  of  Materiel 

and  Nonmateriel  Solutions 

Amphibious  Combat 
Vehicle  ICD  (11- 
151956055-00) 

2011 

Requirements  traceable  to  the  Joint 
Operating  Concept  vice  Universal  Joint 
Task  Lists;  uses  a  typical,  but  good 
example,  of  capability  prioritization/ 
mapping  matrix  (includes  MOEs  and 
Minimum  values);  recommends  mix  of 

Materiel  and  Nonmateriel  Solutions 

Personnel  Recovery 
(12-167465473-00) 

2012 

Succinct  Document;  recommends 

Materiel  and  Nonmateriel  Solutions 

Data  Masked  (12- 
159990107-00) 

2012 

Detailed  analysis  using  several 
techniques;  Well-defined  MOEs 
including  Threshold  and  Objective 

Values 

requirement  definition?  Are  there  characteristics  of  an  ICD  that 
indicate  how  well  a  program  will  adhere  to  cost,  performance,  and 
schedule  expectations? 

Since  2002,  the  JCIDS  process  has  been  refined  and  enhanced.  There 
appears  to  be  a  convergence  in  the  formatting  and  content  of  many  ICD/ 
JCDs  since  2008.  While  the  quality  of  historical  ICDs  varies,  marked 
improvements  to  the  analysis  have  been  documented  since  2008,  pos¬ 
sibly  due  to  the  GAO  report  from  the  same  year. 

Through  research  of  the  current  methodologies  used  in  ICDs  since 
the  inception  of  the  process,  the  research  team  has  formulated  an  out¬ 
line  of  proposed  areas  upon  which  writers  and  implementers  can  focus. 
Future  writers  may  use  this  outline  as  well  as  series  of  DoD  guidelines  to 
provide  the  Joint  community  with  superior  ICDs  that  achieve  their  goals 
in  a  more  efficient  manner  with  minimal  processing  time. 
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APPENDIX 

Additional  Analytical  Techniques  to  Assist  Initial  Capabilities  Document  (ICD) 
Writers  and  Project  Managers 


Method 

Source(s) 

Explanation 

Usage  Context(s) 

Pre-Capabilities-based  Assessment  (CBA) 

Scenario-based 

Planning 

Capabilities- 

based 

Assessment 
(CBA)  User's 
Guide,  p.  87 
(Ringland  & 
Schwartz,  1998) 
(Hiam,1990,  p. 
284) 

Technique  using  scenarios  to  define/give  structure 
to  an  otherwise  murky  strategic  future.  A  type 
of  brainstorming,  which  may  use  nominal  group 
technique  or  another  group  problem-solving 
technique. 

•  Assumptions/drivers  of  change  (identify  key 
variables  and  historical  trends) 

•  Develop  framework  for  drivers 

•  Produce  initial  miniscenarios  (vary  the  type: 
surprise-free,  radical,  and  in-between) 

•  Reduce  to  2  or  3  scenarios 

•  Write  scenarios 

•  Identify  issues  arising  (sensitivity  analysis  with 
scenarios'  impact  on  key  variables) 

Mostly  pre-CBA;  used  to 
build  portfolios;  however, 
can  be  used  in  a  CBA  (e.g., 
to  analyze  threats,  etc.). 

Strengths, 
Weaknesses, 
Opportunities 
and  Threats 
(SWOT)  Analysis 

(Helms  & 

Nixon,  2010) 

Analyzes  internal  (strengths/weaknesses)  and 
external  (opportunities/threats)  factors  to  help 
guide  corporate  strategy  development.  Useful 
in  a  group  strategy  setting,  using  nominal  group 
technique,  or  another  group  problem-solving 
technique  (like  a  Group  Decision  Support  System, 
or  GDSS).  See  also  Porter's  5  Forces  and  Barney's 
Resource-based  View  for  more  specific  analyses. 

Mostly  pre-CBA;  used  to 
build  portfolios;  however, 
can  be  used  in  a  CBA  (e.g., 
to  analyze  threats,  etc.). 
Generallv  criticized  for  its 

lack  of  deoth  and  rigor. 

Porter's  5  Forces 
Analysis 

(Porter,  2008) 

Builds  on  the  "threats/opportunities"  side  of 

SWOT  to  explain  how  market  structure,  defined 
by  five  market  forces  (threat  of  entrants, 
supplier  power,  buyer  power,  intensity  of  rivalry, 
threat  of  substitutes)  and  one  additional  force 
(complementors/government/public)  drive  the 
content  and  performance  of  firms. 

Mostly  pre-CBA;  used  to 
build  portfolios;  however, 
can  be  used  in  a  CBA 
(e.g.,  to  analyze  threats, 
etc.).  Generallv  criticized 
for  focus  on  external 
environment,  vice  internal. 
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APPENDIX  (Continued) 


Method 

Source(s) 

Explanation 

Usage  Context(s) 

Barney's 
Resource-based 
View  (RBV) 

(Barney,  1991) 

Builds  on  the  "strengths/weaknesses''  side  of 
SWOT  to  explain  how  a  firm's  internal  resources 
(value  [V],  rareness  [R],  nonsubstitutability  [NS], 
imperfect  imitability  [//]),  lead  to  sustainable 
competitive  advantage  (SCA).  SCA  =  V  +  R  +  NS 
+  II  Must  have  first  three  to  achieve  comoetitive 

advantage,  and  all  four  to  achieve  SCA. 

Mostly  pre-CBA;  used  to 
build  portfolios;  however, 
can  be  used  in  a  CBA 
(e.g.,  to  analyze  threats, 
etc.).  Generallv  criticized 
for  focus  on  internal 
environment,  vice  external. 

The  Project 
Management 
Diamond 
Approach 

(Shenhar  & 

Dvir,  2007) 

Uses  four  quadrants  of  Technology,  Complexity, 
Novelty,  and  Pace  to  define  the  size,  scope,  and 
risk  of  a  systems  engineering  product/project 

Pre-CBA  (used  to  define  a 
product  portfolio),  CBA/ 
ICD  (developing  Measures 
of  Effectiveness  (MOE), 
Capabilities  Development 
Document  (CDD)  (defining 
system  risk). 

Market 

Segmentation 

Grid 

GAO  Report 

No.  07-388, 

p.11 

A  grid  which  compares  four  markets  (current/ 
new  customers  in  existing  segments/customers 
in  new  segments/new  customer  wants  and 
needs)  to  four  offering  types  (current  business/ 
enhancement  to  current  business/new  business/ 
new  to  industry)  in  order  to  position  portfolio 
projects  into  four  categories  (strike  zone/ 
traditional/pushing  the  envelope/white  space 
opportunity).  A  method  of  analyzing  business 
risk  that  encourages  businesses  to  find  the  right 
mixture  of  categories  of  projects.  Similar  to  Risk/ 
Rewards  Matrix. 

Mostly  pre-CBA;  used  to 
build  portfolios;  however, 
can  be  used  in  a  CBA. 

Risk-rewards 

Matrix 

GAO  Report 

No.  07-388, 

p.  16 

(Hiam,1990,  p. 
377) 

A  grid  that  plots  "risks"  vs.  "rewards"  of  projects. 
Similar  to  Market  Segmentation  Grid  in  that  it 
encourages  businesses  to  find  the  right  mixture 
of  categories  of  projects.  The  same  tool  can  be 
used  to  compare  effectiveness  to  cost  in  the  AoA 
'Alternatives  Comparison"  step  (particularly 
useful  in  showing  confidence  levels  and  threshold 
values).  The  "GE  matrix"  version  of  this  maps 
"business  strength"  (internal)  vs.  "industry 
attractiveness"  (external).  The  circles  may  be 
subdivided  into  market  share/total  market  pies  to 
enhance  anaivsis.  Augments  SWOT. 

Mostly  pre-CBA;  used 
to  build  portfolios; 
however,  can  be  used  in 
a  CBA.  Strength  is  that 
the  confidence  level  of 
estimates  is  captured  (bv 

the  size  of  the  circles). 
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Method  Source(s)  Explanation 


Usage  Context(s) 


Nominal  Group  (Sink,  1983) 
Technique 


A  brainstorming  technique  that  mixes  individual 
and  group  activities  to  attempt  to  increase  the 
amount,  diversity,  and  quality  of  ideas  generated. 
Many  variations,  but  follows  the  basic  process 
below: 

•  Individual  Brainstorming 

•  Sharing  Ideas 

•  Group  Brainstorming  (divergent) 

•  Group  Discussion 

•  Group  Brainstorming  (convergent) 

•  Voting/ranking 


Pre-CBA  strategic 
planning,  CBA  (developing 
capabilities/MOEs), 
Analysis  of  Alternatives 
(AoA)/ICD/CDD 
(developing  attributes/Key 
Performance  Parameters 
[KPP]).  Technique  strong 
in  generating  many 

diverse  ideas  without 

arriving  at  Groupthink. 

Other  group  problem¬ 
solving  techniques  may  be 
superior  (e.g.,  GDSS),  but 

at  an  increased  process 

cost. 


Delphi  Technique  (Goodman, 
1987) 


A  type  of  brainstorming  that  uses  experts  to  a) 
identify  issues  in  their  area  of  expertise,  b)  further 
define  issues  in  their  area  of  expertise,  and  c) 
identify  the  importance  of  issues  in  their  area  of 
expertise.  Generally  uses  3-9  experts,  and  begins 
with  Nominal  Group  Technique,  using  future  rounds 
to  refine/reduce/prioritize  issues. 


CBA  ICD  (capabilities, 
MOEs),  and  AoA/CDD 
(attributes,  KPPs).  An 
example  of  an  "expert" 
systems  analysis 
technique.  Careful  choice 
of  experts  is  essential. 


CBA/ICD 


Capabilities- 

Capabilities- 

1)  Describes  capabilities  required  to  perform  a 

CBA.  Results  in  an  ICD 

based 

based 

mission 

(which  not  only  documents 

Assessment 

Assessment 

2)  Identifies  gaps  in  capabilities  and  associated 

the  CBA,  but  acts  as  a 

(CBA) 

(CBA)  User's 
Guide 

operational  risks 

3)  Establishes  a  requirement  to  address  gaps 

decision  document). 

Initial 

Capabilities 
Document  (ICD) 


Capabilities- 

based 

Assessment 
(CBA)  User's 
Guide 


1)  Describes/summarizes  Concept  of  Operations 
(CONOPS)  (~1  page  explanation  of  CONOPS) 

2)  Describes  guidance  (see  Requirements 
Traceability  Matrix) 

3)  Describes  capabilities  required  (includes 
MOEs/threshold  values) 

4)  Describes  capability  gaps  (prioritized,  if 
possible) 

5)  Summarizes  relevant  threats/operational 
environment 

6)  Proposes  nonmateriel  and  materiel  solutions 
(see  Doctrine,  Organization,  Training,  Materiel, 
Leadership,  Personnel,  Facilities-Policy  [DOTMLPF-P] 
Analysis) 

7)  Final  recommendation  (normally,  but  not 
necessarily,  a  materiel  solution) 


CBA/ICD.  The  ICD  is 
a  decision  document 
to  further  explore  an 
enhanced  capability  (result 
of  a  CBA).  Cornerstone 
document  in  the  Joint 
Capabilities  Integration 
and  Development 
System  (JCIDS)  process. 
Listing  to  the  left  is  not 

comprehensive. 
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APPENDIX  (Continued) 


Method 

Source(s) 

Explanation 

Usage  ContexKs) 

Requirements 

Traceability 

Matrix 

Air  Force 

Instruction 
(AFI)  10-601 

Also  known  as  "house  of  quality,"  traces 
system  attributes  to  operational/user/strategic 
requirements.  Multiple  levels. 

CBA/MOE  (developing 
capabilities/MOEs),  AoA/ 
ICD/CDD  (developing 
system  attributes/KPPs) 

Paired 

Comparisons 

(Blanchard  & 

Fab  rye  ky,  2010, 

p.182) 

To  build  a  rank-ordered  list,  each  of  the  options 
is  presented  to  the  decision  maker  two  at  a  time 
(instead  of  all  at  once).  For  N  criteria  to  be  ranked, 
N(N  -  l)/2  pairs  must  be  compared.  Assumes 
transitivity  of  preferences. 

CBA/MOE/ICD 
(development  of  criteria). 
Rank-orderine  imoortance 

of  design  oarameters/ 

caoabilities/gaDS. 

Porter's  Value 
Chain  Analysis 

(Hiam,1990, 
p,  415) 

(Porter,  1980) 

1)  Select  unit  of  analysis,  both  for  your 
organization  and  for  competitors 

2)  Identify  primary  value-adding  activities  (direct/ 
indirect/quality  assurance) 

Inbound/outbound  logistics,  operations, 
marketing/sales,  service 

3)  Identify  support  activities  (direct/indirect/ 
quality  assurance) 

Procurement,  technical  development,  human 
resource  management,  firm  infrastructure 

4)  Identify  linkages  between  value  chain  activities 

5)  Study  the  value  chain  to  identify  sources  of 
competitive  advantage 

Pre-CBA,  AoA.  Much  like 
a  DOTMLPF-P  Analysis, 
the  value  chain  requires  a 
gap  analysis,  but  not  just 
internal  (between  self  and 
competitors),  and  not  just 
in  isolation  (focus  is  on 
interactions). 

Systems 

Definition  Matrix 

(Sage  & 
Armstrong, 
2000,  p.  98) 

Applies  general  systems  theory  to  define  both  the 
SCOPE  (needs/objectives/criteria)  and  BOUNDS 
(parameters/variables/constraints)  of  a  system 
(e.g.,  capability,  MOEs,  attributes,  KPPs).  Nq_ 
real  analytic  techniaue  used  to  define,  although 
defining  the  SCOPE  and  BOUNDS  of  a  system  can 
use  many  of  the  methods  contained  herein. 

CBA/ICD  (capabilities, 
MOEs),  and  AoA/CDD 
(attributes,  KPPs).  See  also 
Work/Product  Breakdown 
Structure  (WBS/PBS)  for 
a  technique  to  develop  the 
initial  listing  of  attributes. 

Input-Output 

Matrix 

(Sage  & 
Armstrong, 

2000,  p.102) 

Applies  general  systems  theory  to  define  inputs 
(intended/unintended)  and  outputs  (desired/ 
undesired)  and  begin  a  more  sophisticated 
discussion  about  refining  a  system,  such 
as:  situation,  expertise,  risk,  spillover  effects, 
knowledge,  viewpoints,  experience,  kind  of  need, 
frequency,  urgency,  limits,  and  tolerances.  As 
shown  in  Sage,  uses  a  WBS  structure,  but  could 
also  use  a  PBS  structure. 

CBA/ICD  (capabilities, 
MOEs),  and  AoA/CDD 
(attributes,  KPPs). 
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Method 

Source(s) 

Explanation 

Usage  ContexUs) 

Rapid 

Application 

Development 

(RAD) 

(Mackay, 

Came,  Beynon- 
Davies,  & 
Tudhope, 

2000) 

RAD  uses  short,  iterative  design  cycles  to  produce 
working  prototypes  and  systems.  A  mixture  of 
paper  prototypes  (e.g.,  the  different  Department  of 
Defense  Architecture  Framework  (DoDAF)  views, 
use  cases,  screen  shots),  code  stubs  1  menus,  and 
models  may  all  be  used.  Many  types,  including: 

•  Joint  Analysis  and  Design  (JAD):  y2-day  sessions 
placing  developers  and  users  together.  Developers 
use  the  rest  of  the  day  to  build  prototypes.  Lasts 

~1  week. 

•  extreme:  exploration  consists  of  users  writing 
story  cards  (use  case),  which  developers  analyze 
and  give  estimates  to  complete.  Business  then 
prioritizes  the  cards  by  usefulness  and  developers 
prioritize  by  risk.  Best  mix  of  cards  selected  to 
implement. 

Usuallv  used  when 

implementation 

is  more  important 

than  documentation: 

however,  the  process 
of  idea  generation  and 
documentation  makes  this 
technique  ideal  for  pre- 
CBA  and  CBA  activities. 
Technique  may  also  be 
used  in  early  systems 
engineering  (SE)  to  help 
define  systems  (assumes 
that  many  users  do  not 
"know  what  they  want  until 
they  see  it"). 

Use-cases 

Capabilities- 

based 

Assessment 
(CBA)  User's 
Guide,  p.  87 

A  use-case  may  be  as  broad  as  a  story  outlining 
how  a  system  would  be  used  in  an  ideal 
circumstance  (or  multiple  circumstances), 
or  might  be  as  specific  as  a  Unified  Modeling 
Language  (UML)-based  diagram  outlining  a 
specific  system  interaction  that  can  be  used  to 
generate  an  engineering  prototype.  Many  ICDs 
iterate  1-4  possible  "scenarios  for  use,"  with  the 
resulting  scenarios  resembling  SE  use-cases. 

Normally  post-CDD; 
however,  technique  useful 
in  early  SE.  See  Scenario- 
based  Planning  for  a 
similar  technique  applied 
to  large-scale  planning. 

Intelligence- 

based 

Assessment 

Existing  ICDs 

Used  either  to  further  define  a  capability  gap, 
or  to  further  define  the  "threats/operational 
assessment"  category,  this  item  usually  lists 
the  threat  as  defined  by  current  intelligence 
assessments,  as  well  as  the  reference  for  the 
applicable  intelligence  assessment. 

Pre-CBA,CBA,  ICD. 
Analysis  type  is  present  in 
Operations  Plan/Concept 
of  Operations  Plan 
(OPLAN/CONPLAN),  so 
it  helps  trace  operational 
requirements/gaps  to 
those  documents. 

Work/Product 
Breakdown 
Structure  (WBS/ 
PBS) 

(Turner  & 
Cochrane, 

1993) 

May  be  defined  from  top-down  (decomposition), 
or  bottom-up  (engineering).  Begin  with  major 
items,  and  continually  ask  "what  comes  next," 
or  "what  is  this  component/objective  made  of." 

Stop  when  either:  1)  you  know  how  to  measure 
(objectives)  or  2)  a  reasonable  amount  of  work 
(i.e.,  "work  oackase").  "Decomoosition"  risk  is 
that  not  all  end  items  are  identified,  leading  to 

inaccurate  estimate.  Engineering  risk  is  in  omitting 

CBA/MOE  (developing 
criteria),  AoA/ICD/ 

CDD  (system  definition). 
Used  to  decompose 
requirements  or  work 
hierarchically.  May  then 
be  used  for  the  basis 
of  defining/estimating 
work,  cost,  measures  of 

important  integration  items,  or  nonoroduct-related 

tasks  (i.e.,  Proiect  Management.) 

effectiveness,  or  other 
decision  objectives/ 
criteria. 
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APPENDIX  (Continued) 


Method 

Source(s) 

Explanation 

Usage  Context(s) 

Measure  of 
Effectiveness 
(MOE)  Definition 

(Sproles,  2002, 
p,  255) 

•  Request  to  formulate  MOEs 

•  Determine  viewpoint 

•  Determine  mission 

•  Decide  on  Critical  Operational  Issues  (COI),  i.e., 
"tasks/categories") 

•  Draft  MOEs  (creative/testable/consistent  with 
library/statement) 

•  Evaluate/Revise/Agree  on  MOEs 

•  Apply  MOEs 

CBA/ICD.  MOEs  are 
normally  high-level,  and 
one  might  expect  10-20  of 
them  in  an  ICD,  whereas 
a  CDD  might  contain 
hundreds  of  KPPs.  Modern 
ICDs  will  usually  contain 
MOEs  as  well  as  threshold 
values. 

Requirements 
Correlation  Table 

Manual  for  the 
Operation  of  the 
JCIDS,  2012,  p. 
B-31;  AFIIO- 
601,  p.  37 

Summary  of  all  desired  capability  characteristics 
listed  as  threshold/objective  values,  mapped  to 
their  Joint  Capability  Area  (JCA).  Three  tables: 

Key  Performance  Parameters  (KPP),  Key  System 
Attributes  (KSA),  Attribute.  Each  table  has  a  brief 
explanation  of  derivation/justification  of  attributes 
listed. 

•  KPP:  system  attributes  considered  most  critical 
or  essential  for  an  effective  military  capability. 
Failure  to  meet  KPP  threshold  may  result  in 
program  reevaluation/reassessment. 

•  KSA  Table  (AF-only):  Only  the  most  critical 
system  attributes  are  included  and  prioritized. 

•  Additional  Attribute:  Same  as  KSA,  but  contains 
additional  items 

ICD/CDD.  Helps  decision 
makers  and  acquisition 
community  decide  on  most 
important  attributes,  and 
the  threshold  1  objective 
values  those  items  must 
exhibit.  Note  that  JCAs, 
listed  in  the  Manual  for  the 
Operation  of  the  JCIDS,  p. 
B-B-1,  can  be  used  to  assist 
in  attribute  definition  as 
early  as  the  CBA  process, 
as  well  as  to  derive  KPPs 
from  JCAs. 

Capability  Gap 
Matrix 

Existing  ICDs 

Perhaps  the  most  common  table  arising  since 

2008  in  ICDs,  this  table  lists  (in  the  following 
order):  Priority,  Capstone  Concept  for  Joint 
Operations  (CCJO)  Key  Characteristics,  Capability, 
JCAs,  Parameters/Measures  of  Effectiveness, 
and  Minimum  Value  (for  Parameters).  Answers 
many  key  questions,  and  may  be  combined  with  a 
capability  gap  matrix.  See  Requirements  Correlation 
Table  and  Capability  Gap  Pairwise  Matrix. 

CBA,  ICD.  This  table 
combines  capabilities, 
MOEs,  and  minimum 
values.  It  does  not 
directiv  address  caoabilitv 

gaos  (unless  gaos  are 

incoroorated). 
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Method 

Source(s) 

Explanation 

Usage  Context(s) 

Capability  Gap 
Pairwise  Matrix 

Capabilities- 

based 

Assessment 
(CBA)  User's 
Guide,  p.  89; 
Existing  ICDs 

A  method  of  prioritizing  capability  gaps  with 
respect  to  each  other  by  pairwise  comparison 
(using  correlation  matrix).  Each  capability  is 
listed  both  on  the  rows  and  the  columns,  and 
compared  to  others  (1.00  is  "the  same  as,"  while 
0.00-  .99  is  "less  than,"  and  1.01-  >  is  "greater 
than").  The  relative  weight  of  items  to  each 
other  is  multiplicative  (with  2.00  being  "twice 
as  important  as").  Scores  are  summed  across 
rows  (and  normed,  if  desired),  and  then  rank- 
ordered  based  on  the  scores,  with  a  higher  score 
being  more  important.  Note:  One  variation  uses 
"stoplight"  (i.e..  Red,  Yellow,  Green)  to  highlight 
the  degree  to  which  an  attribute  (column) 
represents  a  "gap"  with  current  key  UJTL,  JCA,  etc. 
tasks  (tuple). 

CBA.  Technique  also 
useful  to  rank-order  MOEs 
(ICD)  and/or  criteria 
(AoA/CDD).  See  Pairwise 
Comparison  for  a  similar 
technique  exploring  the 
same  questions  (uses 
transitivity  to  justify  using 
fewer  comparisons). 

Scores,  rankings,  and 

"stoolight"  svmbols  are 

Qualitative  measures, 
assigned  at  the  discretion 

of  the  ICD  team. 

DOTMLPF-P 

Analysis 

Capabilities- 

based 

Assessment 
(CBA)  User's 
Guide;  Manual 
for  the  Operation 
of  the  ICIDS 

Any  analysis  that  includes  the  following  factors 
(and  their  potential  interactions):  Doctrine, 
Organization,  Training,  Materiel,  Leadership 

Policy  and  Education,  Personnel,  Facilities,  and 
Policv.  Imoortant  to  consider  in  all  ohases  of 
earlv  svstems  anaivsis.  including:  a)  Gao  anaivsis 

(CBA),  b)  nonmateriel  solution  (CBA— most 

tvoical  use),  c)  nonmateriel  enablers  to  materiel 

solution  (CBA  and/or  CDD). 

CBA/MOE  (developing 
capabilities/MOEs),  AoA/ 
ICD/CDD  (developing/ 
rating  system  Attributes/ 
KPPs).  See  DOTMLPF-P 
Matrix. 

DOTMLPF-P 

Matrix 

Existing  ICDs 

A  matrix  showing  capability  gaps  and/or 
objectives  down  tuples  and  Y/N/P  answers  to 
DOTMLPF-P  on  each  column,  with  a  "rationale/ 
comments"  column. 

Y  =  gap  may  be  resolved  without  materiel 
development 

N  =  no  solution  currently  exists 

P  =  partial  solution  exists 

CBA/ICD/AoA/CDD. 

This  version  of  the  matrix 

is  tailored  toward  gao 

anaivsis.  soecificallv. 

May  have  other  uses;  see 
DOTMLPF-P  Analysis. 

Cross-interaction 

Matrix 

(Sage  & 
Armstrong, 

2000,  p.110) 

A  correlation  matrix  showing  the  interactions 
between  system  objectives  (as  shown,  uses 
ordinal "+,"  "0,"  and  to  show  interactions, 
but  could  also  use  scalar  Capability  Gap  Matrix 
Measures). 

CBA/ICD  (capabilities, 
MOEs),  and  AoA/CDD 
(attributes,  KPPs). 

Frequency/ 

Investment 

Matrix 

(Williamson, 

1979) 

Recurrent  or  occasional,  but  nonspecific  market 
transactions  are  best  handled  by  classical 
(market)  contracts.  The  tendency  toward 
recurrent  and  idiosyncratic  transactions  tends 
to  favor  unified  governance.  May  also  explain 
boundary  of  firm,  vertical  integration,  and 
departmentalization  (consideration  for  funding 

CBA  work  via  contracts). 

Used  to  determine  type 
of  contract  one  might 
use  to  purchase  different 
types  of  services  on  the 
market.  Uses  transaction 
costs  (immeasurable)  as 

theoretical  mechanism  to 

explain. 
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APPENDIX  (Continued) 


Method  Source(s)  Explanation  Usage  Context(s) 

AoA/CDD 


Analysis  of 

Alternatives 

(AoA) 


AP1 10-601; /AoA 
Handbook, 
pp.  14,  31,  33, 
45,46,47 


The  AoA  is  a  process,  consisting  of  four  basic 
sections:  1)  Effectiveness  Analysis,  2)  Cost 
Analysis,  3)  Risk  Analysis,  and  4)  Alternative 
Comparison.  Each  of  these  four  items  uses 
techniques  such  as  Decision  Evaluation  Matrix  to 
evaluate  alternatives  based  on  MOEs.  MOEs  may 
be  mapped  to  their  overarching  tasks  or  desired 
outcomes. 

•  Effectiveness  Analysis:  1)  Select  Mission  Tasks 
(MT),  MoEs,  and  MoPs,  2)  Select  threats/ 
scenarios,  3)  Describe  alternatives,  4)  Determine 
level  of  detail,  5)  Identify  suitable  analysis  tools/ 
data  sources  (consider  including  sensitivity 
analysis) 

•  Cost  Analysis:  1)  sunk,  2)  research  and 
development,  3)  investment,  4)  operating/ 
support,  5)  disposal,  6)  baseline  extension,  7) 
prefielding 

•  Risk  Analysis:  see  Risk  Analysis 

•  Alternative  Comparison:  see  Decision  Evaluation 
Matrix  and  Risk-Rewards  Matrix.  The  AoA  Handbook 
shows  a  Decision  Evaluation  Matrix  with  additional 
columns  (for  risk  and  cost). 


AoA/ICD  (developing 
and  applying  MOE 
to  capabilities),  CDD 
(developing  and  applying 
criteria  to  alternatives). 

The  items  used  to  BOUND 

the  AoA  are  same  items 

used  to  BOUND  the  ICD. 

AoA  Handbook  gives 
guidelines  for  performing 

the  steps,  overview 

of  analysis  tools,  and 

modeling  suggestions. 

Finally,  AoAs  need  not 

identify  a  single  solution 

(in  fact,  they  may  identify  a 

suite  of  solutions  that  meet 

certain  requirements). 


Decision 

Evaluation 

Display 


(Blanchard  & 
Fabrycky,  2010, 
p.187) 


Graphical  representation  of:  1)  alternatives  (A,  B, 
C);  2)  equivalent  cost/profit,  3)  other  criteria  (X, 

Y,  Z).  Although  not  strictly  a  2-dimensional  view, 
the  x-axis  is  structured  according  to  increasing 
cost/profit  of  alternatives,  and  the  y-axis  is  scaled 
with  relative  (ordinal,  i.e.,  less  than,  equal  to, 
more  than)  achievement  by  alternatives  of  the 
criteria.  (Note:  Normally,  these  would  be  separate 
graphs  for  each  criteria,  but  they  are  stacked  on 
top  of  each  other  to  simplify  the  display,  with  no 

implication  of  relevance  of  the  different  position  of 

each  criterion  on  the  v-axis  fexcept  with  reference 

to  itselfT) 


CBA/MOE  (developing 
criteria),  AoA/ICD/ 

CDD  {applying  criteria). 
Organizes  information  on 
alternatives  and  degree  of 
compliance  with  criteria 
(including  threshold 
values)  while  still  allowing 
for  decision-maker  insight, 
intuition,  and  judgment. 
Not  intended  to  be 
mathematically  applied. 
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Method  Source(s)  Explanation 


Usage  Context(s) 


Decision 

Evaluation 

Matrix 


Decision  Tree 


(Blanchard  &  A  matrix  with  alternatives  on  the  x-axis  (as  a 
Fabrycky,  2010,  tuple),  and  three  items  on  the  y: 
p.  189)  •  Header  #1:  a  future  not  under  the  control  of  the 

decision  maker  ("state  of  nature") 

•  Header  #2:  the  probability  (p)  of  that  future 

•  Each  cell:  evaluation  (E)  measure  (positive 
or  negative)  of  [alternative  x  future];  may  be 
subjective  (i.e.,  categorical)  or  objective  (e.g., 
monetary  values). 

Possible  decision-making  criterion  (to  select  most 
desirable  alternative): 

•  Aspiration  level:  setting  desired  min  and  max 
levels  for  each  criterion,  or  for  all  criteria  as  a 
whole 

•  Most  probable  future:  useful  if  one  probability 
dominates 

•  Expected  value  (EV):  EV  =  Z(E  X  p)  where  Zp  = 
1.00  useful  for  repetitive  environment 

~  Laplace:  if  p  unknown  for  each  alternative, 
divide  1.00  by  number  of  alternatives) 

~  Maximin:  best  alternative  given  the  worst 
possible  outcome 

~  Maximax:  best  alternative  given  the  best 
possible  outcome 

~  Minimax  (includes  "regret"):  best  outcome 
-  outcome  for  "a."/"s.";  attempts  to  calculate 
opportunity  cost  of  a  decision 
~  Hurwicz  rule:  assigns  an  optimism  index  from 
0-1.0  (assumes  linearity) 


CBA/MOE  (developing 
criteria).  AoA/  CD/ 

DD  (applying  criteria). 
Considers  alternatives/ 
criterion  of  effectiveness 
in  past/present,  but  also 
alternatives/possible 
future  conditions  [of 
use].  Assumes:  all  viable 
alternatives  considered, 

all  possible  futures 

identified,  all  futures 

and  [alternatives  x 
futures]  are  orthogonal, 

occurrence  of  specific 

future  is  unknown 

(otherwise,  matrix 
simplifies  to  a  vector  of 

evaluation  measures). 

Limitation:  each  of  these 

methods  yields  different 

results. 


(Kirkwood, 

2002) 


Calculates  an  expected  value  (EV)  for  each  of 
a  number  of  possible  options,  exploring  what 
happens  if  selection  leads  to  success  or  failure. 
May  include  a  "none  of  the  above"  option.  One 
common  use  is  to  include,  add  together  the  cost  of 
each  of  the  options  with  their  expected  payout  to 
generate  the  evaluation  (E)  measure. 

•  EV  =  Z(E  X  p),  where  Zp  =  1.00  for  the  outcome 
of  each  decision.  Most  useful  for  repetitive 
environment:  otherwise,  the  EV  metric  has  no 

inherent  meaning  (although  often  shown  as 

monetary  value,  $). 


CBA/CD/AoA/CDD. 
Amenable  to  monetary 
decisions  that  can 
be  stated  in  Boolean 

(success/failure) 
terms.  Options  must  be 

orthogonal.  May  also  be 

used  to  model  multiple, 

sequential  decisions. 

See  Decision  Evaluation 
Matrix  for  an  additional 
application  of  this 
technique. 
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APPENDIX  (Continued) 


Method  Source(s)  Explanation 


Usage  Context(s) 


Optimization 

Modeling/Linear 

Programming 


(Blanchard  & 
Fab  rye  ky,  2010, 
p.177) 


E  =  f(X,  y.) 

X  =  Design  variables  (factors  that  define  design 
optimization  space) 


=  Design-dependent  variables  (under  control  of 
designers) 


Y.  =  Design-independent  variables  (not  under 
designer-control) 


CBA/MOE  (developing 
criteria),  AoA/ICD/ 

CDD  (applying  criteria). 
Determining  effectiveness 
of  a  system  based  on  a 
model  of  that  system 
including  the  most  relevant 

variables.  Models  lack  of 
certainty  due  to  factors  not 
under  designers'  control. 
See  Decision  Evaluation 
Matrix  for  an  additional 
application  of  this 
technique. 


"Scorecard"  (Sage  &  Yet  another  technique  to  compare  alternatives  to  CBA/ICD  (capabilities. 

Matrix  Armstrong,  criteria,  this  time  with  the  emphasis  on  technology  MOEs),  and  AoA/CDD 

2000,  p.  Ill)  maturity  alternatives  (see  Market  Segmentation  (attributes,  KPPs). 

Grid)  crossed  with  the  "-ilities"— although  any 
combination  thereof  with  other  techniques  in  this 
listing  could  be  used. 


Utility 

(Indifference) 

Curves 


(Brosh,1985, 

p.70) 


Having  developed  a  decision  tree  with  monetary 
outcomes  (but  not  yet  assigned  probabilities  of 
outcomes),  it  is  possible  to  query  the  decision 
maker  as  to  the  amount  deemed  acceptable 
as  a  guaranteed  payout  instead  of  accepting 
the  probabilities  of  payouts  represented  in  the 
decision  tree.  Varying  the  probabilities  and  re¬ 
asking  this  question  allows  one  to  create  a  utility 
curve,  with  the  payout  on  the  x-axis  and  utilities 
on  the  y-axis.  The  "risk-neutral"  decision  maker's 
utility  curve  is  negative  first  derivative  (positive, 
but  decreasing),  while  the  "risk-averse"  is  a 
positive  first  derivative  (positive,  but  increasing). 


Answers  question  of 
decision  maker's  risk- 
averse/neutral/seeking 
nature,  i.e.,  is  valuation  of 
marginal  utility  of  money 
decreasing/constant/ 
increasing?  Determines 
whether  to  use  minimin, 
minimax,  maximax. 

Paired  with  Decision 
Evaluation  Matrix  to  model 
alternative  preference 
in  terms  of  "utility"  vice 
"monetary." 


Weighting  (Blanchard  &  Weights  (W)  must  sum  to  1.00  (100%)  for  each 

Fabrycky,  2010,  criterion.  Ratings  (R)  based  on  whatever  scalar 
p.  185)  rating  schema  one  devises  (does  not  work  for 

ordinal/categorical  ratings). 

Weighted  Rating  =  Wx  R 

•  Tabular  display:  results  indicate  how  close  each 
alternative  comes  to  the  ideal. 

•  Graphical  additive:  results  indicate  the  overall 
contribution  of  the  rating  in  each  category  to  the 
overall  desirability  of  the  alternative. 


CBA/MOE  (developing 
criteria),  AoA/ICD/ 

CDD  (applying  criteria). 
Choosing  across  a  number 
of  design  alternatives  when 
categories  are  not  of  equal 

importance  (see  systematic 
elimination  for  another 
similar  method).  Caution  is 
advised  in  developing  both 

criterion  and  weighting,  as 

well  as  in  interpreting  two 

alternatives  that  end  up 


rating  near  each  other  on 

the  scale. 
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Method  Source(s)  Explanation 


Usage  Context(s) 


Z-score 

Transformation 


(Daszykowski, 
Kaczmarek, 
Vander  Heyden, 
Sl  Waiczak, 
2007) 


For  items  collected  using  ratio/continuous  data  for  AoA. 
which  an  expected  value  (mean)  and  dispersion 
(standard  deviation)  are  known,  application  of  a 
z-transformation  can  re-score  an  item  (results  in 
a  number  between  -1.0  and  +1.0).  Items  can  then 
be  further  transformed  by  weighting  or  another 
technique  and  be  comparable  across  different  items 
(e.g.,  "time-to-implement"  vs.  "distance"). 


National  Family  (Hiam,  1990,  p. 
Opinion  (NFO)  273) 

Product  Analysis 


1)  Survey  customer  attitudes  to  obtain  rankings  of 
importance  of  product  attributes  and  a  rating  of 
the  overall  product  (Likert-type  scale:  5-point,  -2 
to  +2).  Likert  scales  are  commonly  used  in  surveys 
to  measure  attitudes. 

For  example: 

My  current  level  of  job  satisfaction  is: 

1  2  3  4  5  6  7 

Extremely  Extremely 

Unsatisfied  Satisfied 


AoA.  A  method  like  this 
compares  the  perceived 
"gap  severity"  with 
the  "importance"  of 
an  attribute  in  order  to 
assist  the  researcher  in 
prioritizing  the  attributes. 


or  in  the  case  of  the  NFO  Product  Analysis: 

How  well  does  the  product  meet  the  desired 
attribute  (x)? 

-2  -1  0  +1  +2 

Not  nearly  Far  too 

enough  of  x  much  ofx 


2)  Use  stepwise  linear  regression  to  determine  most 
important  attributes  to  overall  ratings  (calculate 
then  "Importance  Index":  R^i^d/ 

3)  Graph  the  "importance  index"  vs.  mean  ratings 
for  each  attribute.  Items  on  upper  corners  are 
those  worth  investing  effort  into. 


Systematic 

Elimination 


(Blanchard  & 
Fab  rye  ky,  2010, 
p.183) 


Do  not  consider  weights,  nor  trade-offs  across 

alternatives.  May  use  scalar  or  categorical  ratings. 

•  Compare  alternatives  against  each  other  (norm- 
referencing;  will  establish  dominance  between  two 
options  [drop  the  lower  one]). 

•  Compare  alternatives  against  a  standard  (criterion- 
referencing:  1)  retaining  if  meets  standard  for  at  least 
one  criterion,  or  2)  retaining  if  meets  standard  for  all 
criterion). 

•  Comparing  criteria  across  alternatives  (after  ranking 
criterion:  1)  choose  best  alternative,  break  ties  with 
the  second  most  important  criterion,  or  2)  examine 
one  criterion  at  a  time,  comparing  the  alternatives  and 
eliminate  those  not  meeting  minimum  standard). 


CBA/MOE  (developing 
criteria),  AoA/ICD/ 

CDD  (applying  criteria). 
Choosing  across  a  number 
of  design  alternatives. 
Outcomes  can  be 
specified  for  all  criteria 

and  alternatives.  May 
use  to  select  best  option, 
or  to  determine  which 
of  a  number  of  options 
meet  minimum  criteria 
for  further  inclusion.  See 
weighting  for  another 
similar  method. 
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APPENDIX  (Continued) 


Method 

Source(s) 

Explanation 

Usage  Context(s) 

Sahid's 

Consequences 

Table 

(Hammond, 
Keeney,  & 

Raiffa,  1998) 

Lists  alternatives  across  the  columns  and  key 
attributes/decision  criteria  down  tuples.  The 
goal  of  this  table  is  not  to  combine  disparate 
data  types,  but  rather  to  search  for  options  that 
clearly  "dominate"  other  options.  The  "dominated" 
options  are  then  eliminated  systematically. 

AoA.  Because  it  is  an 
initial  screening  process,  it 
reduces  options/simplifies 
choice;  however,  ensure 
the  most  imoortant 
attributes  are  screened 

first. 

Even  Swaps 

(Hammond, 
Keeney,  & 

Raiffa,  1998) 

A  more  sophisticated  analysis  using  Sahid's  Table, 
"how  much  of  one  attribute  are  you  willing  to 
swap  for  an  increase/decrease  in  the  other?"  In 
this  way,  attributes  of  key  interest  can  be  made 
comparable  by  trading  up/down  other  attributes. 
This  is  one  form  of  sensitivity  analysis. 

AoA.  Does  not  treat 
alternatives  as  exclusive: 

encourages  decision  maker 

to  look  for  (not  listed) 

alternatives  to  satisfv 

"swaoDed"  items. 

Risk  Analysis 

AoA  Handbook, 
p.  40 

Risks  are  categorized  by  Severity  (S,  i.e., 
consequence)  and  Probability  (p,  e.g.,  likelihood). 

If  each  risk  is  assigned  a  number  from  0.00  - 1.00 
for  both  categories,  then  a  composite  risk  index 
can  be  calculated  using:  CR  =  Sxp,  and  a  risk 
matrix  can  be  used  to  plot  the  results.  Risk  may 
then  be  avoided,  accepted,  transferred,  and/or 
mitigated.  Some  add  three  columns  to  a  risk  table 
to  add  how  the  risk  was  managed,  the  resultant 
risk,  and  any  secondary  risks  that  risk  mitigation 
created. 

AoA.  Technique  uses 
qualitative  assignment  of 
risk  values.  Normallv.  risks 
are  assumed  orthogonal 

(however,  risk  interactions 

can  be  modeled  with  this 

techniaue). 

Sensitivity 

Analysis 

(Blanchard  & 

Fab  rye  ky,  2010, 
pp.  589,  614) 

A  generic  category  of  tools  that  plots/graphs/ 
calculates  the  relationship  between  changing 
variables,  giving  an  idea  of  how  a  modification 
in  one  variable  affects  others.  Plotting  different 
alternatives  on  the  same  axis  gives  an  idea  of 
the  favorability  of  one  option  versus  the  other  in 
the  trade  space  measured  (a.k.a.  the  "breakeven 
point").  Examples:  Pareto  chart  (a  line  or  bar 
graph  displaying  results  ordered  by  frequency  of 
occurrence),  scatter  plot,  cost/year  plot. 

Primarily  AoA/CDD,  but 
can  be  used  in  CBA/ICD. 

Cost  Breakdown 
Structure  (CBS) 

(Blanchard  & 

Fab  rye  ky,  2010, 
p.  577) 

Similar  to  a  WBS/PBS,  a  CBS  breaks  all  costs  down, 
either  by  product,  cost  center,  or  development 
phase.  Blanchard  and  Fabrycky  call  this  a  "functional" 
breakdown).  A  typical  CBS  might  include  items  such 
as:  research  and  development  cost,  production/ 
construction  cost,  operations  &  maintenance  cost, 
retirement  and  disposal  cost.  Many  of  the  cost 
categories  included  in  a  CBS  are  standardized  items 
in  the  finance  community,  and  each  has  estimation 
technique(s)  associated  with  it.  Costs  are  often 
captured  on  a  Cost  Collection  Worksheet. 

AoA/CDD.  The  U.S. 
militarv  does  not  normallv 

perform  some  of  the  kev 

items  included  in  a  CBS; 

therefore,  estimates  in 

these  areas  mav  not 

be  reliable  (or  else  the 
militarv  might  contract  the 

cost  estimate). 
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Method 

Source(s) 

Explanation 

Usage  Context(s) 

Cost  Collection 
Worksheet 

(Blanchard  & 

Fab  rye  ky,  2010, 
p.586) 

Basic  mechanism  used  to  gather  and  report  costs 
generated  by  a  CBS.  Much  like  a  WBS,  costs  are 
broken  down  by  function  and  subfunction  (and 
the  associated  cost  categories)  in  the  tuples,  while 
the  cost  by  program  year,  total  (actual),  total  net 
present  value  and  %  contribution  are  in  columns. 

AoA/CDD.  Compares 
programs  by  cost  center/ 
year,  or  cost  profile  (since 
profile  by  center/year  is 
accessible  to  viewing). 

Parameter-based 

Costing 

(Blanchard  & 
Fabrycky,  2010, 
p.581) 

One  of  the  four  types  of  cost  estimating, 
parametric  analysis,  involves  determining  key 
parameters  that  drive  cost  (historically),  then 
using  these  parameters  to  estimate  future  costs. 

AoA/CDD.  Onlv  as  eood 
as  oast  information  and 

current  iudement. 

Activity-based 

Costing 

(Blanchard  & 
Fabrycky,  2010, 
p.581) 

A  method  directed  toward  "detailing  and 
assignment  of  all  costs  to  the  activities  that  cause 
them  to  occur,"  in  an  effort  to  include  traceability 
(for  items  historically  difficult  to  track;  i.e.,  indirect 
costs  like  "overhead"). 

AoA/CDD.  Mav  be  at  odds 
with  WBS/PBS  methods 

of  tracking  costs  (because 

functions  like  oroiect 

management  spread 

across  multiole  cost 

centers). 

Life-cycle  Cost 
Summary 

AoA  Handbook, 
p.37 

Breaks  out  life-cycle  costs  two  ways:  1)  by 
alternative  and  life-cycle  phase,  2)  by  budget 
category  and  life-cycle  year  (any  combination  of 
these  is  acceptable,  based  on  the  requirement). 

AoA. 

Money  Flow 
Modeling 

(Blanchard  & 
Fabrycky,  2010, 
p.176) 

Considers  present  equivalent  (PE),  annual 
equivalent  (AE),  or  future  equivalent  (FE)  amount, 
as  well  as  internal  rate  of  return  and  payback 
period. 

PE,  AE,  or  FE  =  f(F^,  /,  n) 

t  =  0,1,2, ... ,  n  (salvage  value/cost  added  at  end  of 
final  year) 

F^  =  positive  or  negative  money  flow  at  end  of  year  t 

ICD/AoA/CDD  (economic 
analysis  of  alternatives). 
Calculating  outlay  and 
payback  of  a  system 
over  its  acquisition  and 
utilization.  See  Decision 
Evaluation  Matrix  for  an 
additional  application  of 
this  technique. 

1  =  annual  interest  rate 

n  =  number  of  years 
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A  Proposed  2025  Ground 
Systems  “Systems  Engineering” 

Process 


Robert  E.  Smith  and  LTC  Brian  D.  Vogt,  USA 


The  U.S.  Army’s  mission  reflects  a  strong  impetus  to 
provide  flexible  and  adaptable  ground  vehicles  that  are 
rapidly  fieldable.  Emerging  manufacturing  technology, 
such  as  three-dimensional  (3D)  printing,  is  making  mass 
customization  possible  in  commercial  industry.  If  the 
Army  could  produce  tailored  military  ground  vehicles 
that  incorporate  mission-specific  tactics,  it  v\/ould  outper¬ 
form  generic  systems.  To  produce  such  systems,  a  new 
systems  engineering  (SE)  process  should  be  developed. 
Virtual  environments  are  central  to  the  proposed  SE/2025 
process  because  they  provide  a  sandbox  where  soldiers 
and  engineers  might  directly  collaborate  to  codevelop 
tactics  and  technologies  simultaneously.  The  authors’ 
intent  is  to  describe  how  ground  vehicle  systems  might 
be  developed  in  2025  as  well  as  to  describe  current 
efforts  underway  to  shape  the  future. 
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In  the  past,  the  United  States  Army  has  been  able  to  anticipate  capa¬ 
bility  gaps  and  needs  based  on  a  relatively  static  threat,  but  that  model 
has  disintegrated  over  the  past  two  decades  (United  States  Army,  2013). 
Figure  1  illustrates  pictorially  the  range  and  complexity  of  the  current 
defense  landscape.  Constantly  shifting  mission  requirements  will  likely 
remain  the  norm  in  the  foreseeable  future.  As  such,  combatant  com¬ 
manders  will  need  ground  vehicles,  including  robots  that  are  flexible, 
adaptable,  and  rapidly  deployable.  Additionally,  some  of  the  most  prom¬ 
ising  future  warflghting  technologies,  such  as  robotics,  computing,  and 
advanced  communications,  will  be  readily  available  for  non-State  actors 
and  nations  to  purchase  from  the  global  commercial  market.  To  maintain 
a  military  advantage,  the  United  States  needs  to  develop  a  process  that 
enables  the  lucid  and  rapid  production  of  mission-tailored  platforms  that 
do  not  rely  solely  on  cutting-edge  technology.  Just  as  radar  stealth  and 
drones  were  game  changers  in  the  past,  the  acquisition  process  itself 
could  become  a  game-changing  technology  in  the  future. 

The  Department  of  Defense  (DoD)  acquisition  process  transforms 
warflghter  needs  into  materiel  by  three  separate,  but  interlinked  pro¬ 
cesses:  the  Joint  Capabilities  Integration  and  Development  System; 
the  Planning,  Programming,  Budgeting  and  Execution  System;  and  the 
Defense  Acquisition  System.  According  to  Chyma  (2010),  these  processes 
answer  four  basic  questions: 

•  What  is  the  requirement? 

•  What  is  the  acquisition  strategy? 

•  What  is  the  cost  estimate? 

•  Is  it  affordable? 

The  current  process  is  linear  and  document-centric,  which  makes 
the  process  of  answering  these  questions  in  an  integrated  manner  very 
challenging.  According  to  Boehm  (2010),  "The  weakest  link  in  systems 
engineering  is  often  the  link  between  what  the  warflghters  need  and 
what  the  development  team  thinks  they  need,  together  with  a  shared 
understanding  of  the  operational  environment  and  associated  con¬ 
straints  and  dependencies”  (p.  20). 
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Systems  Engineering  2025  (SE/2025),  as  described  in  this  article, 
explores  a  possible  future  process  to  address  shortfalls  in  the  interlinked 
acquisition  processes  by  using  virtual  worlds  to  enable  new  levels  of  col¬ 
laboration  and  experimentation  with  changeable  tailored  platforms.  The 
year  2025  represents  a  symbolic  point  in  time  where  rapid  manufactur¬ 
ing  will  start  to  provide  the  ability  to  produce  systems  effectively.  The 
authors’  intent  is  to  outline  how  ground  systems  might  be  developed  in 
2025  as  well  as  to  describe  current  efforts  underway  to  shape  the  future. 

Figure  2  shows  the  SE/2025  process  flow.  The  entry  point  into  the 
process  starts  with  the  Persistent  Synthetic  Gaming  Environments  (left 
center)  where  thousands  of  soldiers  may  “kick  the  tires”  on  technologies 
and  customize  vehicles.  This  game-based  environment  will  also  provide 
a  discussion  group  where  soldiers  can  pool  their  collective  expertise  and 
brainstorm  solutions.  Meanwhile,  engineers  can  observe  what  is  working 
and  program  managers  can  assess  the  true  tactical  value  of  technolo¬ 
gies  versus  cost.  Real-time  scenarios  can  be  created  for  experimentation 

FIGURE  2.  GROUND  SYSTEMS  SE/2025  “SYSTEMS 

ENGINEERING”  PROCESS. 
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by  using  intelligence  assets  to  create  instantaneous  geo-specific  envi¬ 
ronments  as  shown  in  the  upper  left  of  Figure  2.  To  avoid  overwhelming 
users  with  choices  from  the  infinite  combination  of  vehicle  technologies, 
vehicle  templates  and  capability  modules  will  be  evolved  within  the  gam¬ 
ing  environments  as  shown  at  the  lower  left  of  Figure  2.  Vehicle  templates 
are  preferred  configurations  of  modules  and  technology  that  the  crowd  of 
soldier-gamers  proves  to  be  robust  for  mission  effectiveness.  The  templates 
will  adapt  over  time  as  users  share  among  themselves  and  piggyback  on 
the  best  ideas.  The  overarching  theme  is  that  a  tailored  system  will  nearly 
always  outperform  a  standardized  system  that  tries  to  do  everything. 

While  not  explicitly  illustrated  in  Figure  2,  a  critical  feature  for  the 
success  of  SE/2025  is  enhanced  communication  between  stakeholders 
across  the  acquisition  community.  Korfiatis  &  Cloutier  (2013)  showed  the 
promise  of  immersive  environments  (especially  gaming  environments) 
to  facilitate  a  deeper  understanding  of  CONOPs  (Concept  of  Operations) 
by  immersing  the  team  in  an  experiential,  first-person  environment.  To 
further  maximize  communications  effectiveness,  information  should  be 
provided  at  just  the  right  time  in  a  format  or  dashboard  that  allows  quick 
interpretation  of  complex  data  and  that  hides  irrelevant  details.  A  recent 
emergence  is  the  employment  of  tradespace  exploration  tools  by  both  the 
Army  Whole  System  Trades  Analysis  (WSTAT)  (Edwards,  2012)  and  the 
Marine  Corps  Framework  for  Assessment  of  Cost  and  Technology  (FACT) 
(Browne,  Ender,  Yates,  &  O'Neal,  2012).  FACT  and  WSTAT  are  both  excel¬ 
lent  examples  of  how  SE  is  beginning  to  provide  dashboard  information  to 
decision  makers.  These  tools  allow  highly  visual  and  interactive  explora¬ 
tions  of  the  tradespace,  which  would  otherwise  be  extremely  challenging 
to  achieve.  Employing  modular  designs  will  also  help  communications 
because  modules  are  essentially  black  boxes  that  will  only  need  to  be  dealt 
with  at  their  interfaces. 

The  final  section  of  SE/2025  in  Figure  2  is  Manufacture  and 
Deployment.  Manufacturing  and  Logistics  will  likely  become  inseparable 
in  the  future  as  localized  production  and  rapid  manufacturing  have  the 
potential  to  become  the  norm.  The  Army  will  find  itself  with  new  choices 
as  to  what  is  produced  stateside,  regionally,  and  at  forward  operating  bases 
(FOB).  True  capability- on- demand  will  be  realizable  when  rapid  manufac¬ 
turing,  and  plug-and-play  modular  components  enable  mass  customization. 
Already,  the  Henry  Ford-era  mass  production  paradigm  is  eroding  within 
the  automotive  industry  where  high  levels  of  customization  are  increasingly 
available  in  the  marketplace  (White,  2012;  Muller,  2010). 
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Deciding  what  is  produced  stateside,  regionally,  and  at  FOBs  will 
depend  on  the  portability  of  manufacturing  equipment,  nature  of  modu¬ 
larity,  and  deployment  timeframes.  Items  that  require  large  amounts  of 
energy,  materials,  and  specialized  environments  (like  clean  rooms)  will 
likely  be  produced  stateside.  In  contrast,  some  vehicle  components  might 
be  digitally  e-mailed  to  an  FOB  and  produced  expeditionary  on  site.  A 
large  benefit  of  3D  printing,  also  known  as  additive  manufacturing,  is 
that  it  takes  a  generic  base  material  such  as  a  powdered  metal  and  fuses 
it  layer  by  layer  into  a  final  piece.  This  means  one  machine  and  one  base 
material  can  produce  a  quite  varied  set  of  components.  The  need  to  be 
rapidly  deployable  will  drive  designs  toward  kittable  solutions  to  mini¬ 
mize  the  initial-entry  airlift  weight.  Armor  kits,  for  example,  can  then 
be  applied  later.  Soldiers  may  swap  modules  on  and  off  vehicles  in  the 
field— just  like  assembling  Lego  toys— to  provide  a  rapid  observe,  orient, 
decide,  and  act  loop  (Boyd,  1996). 

DoD  SE  Process  Versus  the  Competition 


Presently,  we  are  competing  against  the  business  models  of  terror¬ 
ists  and  insurgents  (and  many  threat  countries)  which  are  “very  much 
agile  and  open  approach.  They  do  not  have  thick  internal  R&D  [research 
and  development]  establishments,  and  are 
willing  to  take  knowledge  and  technolo¬ 
gies  from  anywhere  to  achieve  their  goals” 

(Hood,  2007).  Additionally,  insurgents 
have  made  excellent  use  of  the  Internet 
for  collaboration  and  knowledge  sharing. 

They  engage  in  rapid  development  and 
agile  systems  engineering  through  real- 
world  application.  Army  General  James 
Cartwright,  former  vice  chairman  of 
the  Joint  Chiefs  of  Staff,  as  quoted  by 
Kitfield  (2013),  states,  “...  if  you  take 
the  hunt  for  lED  [improvised  explosive 
device]  cells,  that  was  a  30-day  fight.” 

The  enemy  would  invent  a  fuse,  U.S. 
forces  would  develop  a  counter  to  it,  and 
the  enemy  would  respond  by  inventing 
another  triggering  device.  “And  if  it  took 
you  longer  than  30  days  to  respond  to  a  change  in 
enemy  tactics,  your  people  were  dying.”  The  United 
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States  Army  needs  to  shorten  its  materiel  acquisition  observe,  orient, 
decide,  and  act  loop  (Boyd,  1996)  to  keep  a  decisive  advantage  over  an 
innovative  asymmetric  enemy. 

The  current  DoD  process  is  a  linear,  requirements-first  system  in  the 
translation  of  user  needs  into  materiel  solutions  (Boehm,  2010).  Due  to 
the  length  of  the  existing  process,  decisions  made  and  available  technolo¬ 
gies  that  were  relevant  at  the  beginning  of  a  program  may  be  obsolete 
by  the  end  of  the  program.  To  quote  the  Chinese-authored  Unrestricted 
Warfare,  which  discusses  how  developing  countries  might  counter  the 
United  States,  “Customizing  weapons  systems  to  tactics  which  are  still 
being  explored  and  studied  is  like  preparing  food  for  a  great  banquet 
without  knowing  who  is  coming,  where  the  slightest  error  can  lead  one 
far  astray”  (Liang  &  Xiangsui,  1999). 

Liang  &  Xiangsui  (1999)  further  explore  the  fact  that  the  United 
States  generates  a  vast  amount  of  technology  on  which  it  has  been  unable 
to  capitalize,  pointing  out  that: 

...proposing  a  new  concept  of  weapons  does  not  require  relying 
on  the  springboard  of  new  technology,  it  just  demands  lucid 
and  incisive  thinking.  However,  this  is  not  a  strong  point  of  the 
Americans,  who  are  slaves  to  technology  in  their  thinking.  The 
Americans  invariably  halt  their  thinking  at  the  boundary  where 
technology  has  not  yet  reached,  (p.  24) 

Development  of  the  first  crowdsourced  military  vehicle,  the 
Flypmode,  by  the  Defense  Advanced  Research  Projects  Agency  (DARPA) 
and  Local  Motors  gives  a  glimpse  of  the  potential  for  SE/2025.  Jay 
Rogers,  founder  of  Local  Motors,  points  out  conflicts  are  won  not  by 
spending  tons  of  time  and  billions  of  dollars,  but  “They  win  it  because 
they  figured  out  what  was  going  to  beat  the  enemy,  and  they  built  that” 
(Boyle,  2011).  Rogers  went  on  to  say: 

Maybe  we  did  not  do  the  same  development  that  [the  contractor] 
did,  to  make  sure  the  strut  on  the  vehicle  lasts  a  million  miles. 
But  if  it  saves  a  life,  and  it  lasts  for  a  whole  conflict,  haven't  we 
done  abetter  thing? 

President  Barack  Obama  was  shown  the  Flypmode  vehicle,  which  only 
took  4  months  to  produce  (Boyle,  2011),  and  enthusiastically  pointed  out: 
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Not  only  could  this  change  the  way  the  government  uses  your 
tax  dollars— think  about  it,  instead  of  having  a  10-year  lead  time 
to  develop  a  piece  of  equipment,  if  we  were  able  to  collapse  the 
pace  of  which  that  manufacturing  takes  place,  that  would  save 
taxpayers  billions  of  dollars— but  it  also  could  get  technology  out 
to  the  theater  faster,  which  could  save  lives. 

Persistent  Synthetic  Gaming  Environments  (Soidier 
Crowdsourcing) 

The  use  of  video  games  is  not  new  to  the  Army.  In  1981,  General  Bonn 
A.  Starry,  then-commander  of  the  U.S.  Army  Training  and  Doctrine 
Command,  was  struck  by  what  he  saw  in  the  video  game  arcades 
(Trachtman,  1981): 

I  see  a  lot  of  people  in  those  arcades  learning  something,  and 
they're  all  volunteers,  and  they're  paying  a  quarter  to  learn  what¬ 
ever  it  is  they  learn  from  these  machines.  I  don't  know  what  they 
learn,  but  I'm  convinced  they  learn  something,  and  that  the  Army 
needs  to  exploit  it.  (p.  56) 

SE/2025  proposes  to  tap  into  thousands  of  soldiers,  who  already  play 
video  games  in  their  spare  time. 

The  Army  Capabilities  Integration  Center  (ARCIC)  has  begun  an 
experiment  called  Early  Synthetic  Prototyping  to  create  a  persistent 
gaming  environment  to  answer  the  following  question: 

How  does  the  Army  develop  and  implement  a  process  and  a  set  of 
tools  that  enables  soldiers  to  assess  emerging  technologies  in  a 
synthetic  environment  to  provide  relevant  feedback  that  informs 
science  and  technology  research,  doctrine,  organization,  and 
training  development? 

Past  game-based  experiments  were  not  persistent  and  were  limited 
in  participation  to  a  relatively  small  user  base.  The  target  of  the  ARCIC 
investigation  is  to  involve  upwards  of  a  thousand  soldiers  in  the  gam¬ 
ing,  which  is  crowdsourcing.  However,  open  research  questions  remain 
unanswered  about  this  methodology,  including: 
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•  Can  we  draw  (explicitly  and/or  implicitly)  useful  feedback 
from  soldiers  about  future  technology  capabilities  using  a 
game  environment  as  a  concrete  experience? 

•  Are  the  results  of  analysis  from  soldier  feedback  signifi¬ 
cantly  different  from  the  results  of  analysis  from  traditional 
experimentation? 

•  How  do  we  begin  to  allow  soldiers  an  active  role  in  the 
design  of  platforms? 

Research  is  presently  being  conducted  by  the  authors  to  answer  these 
questions. 

The  fundamental  purpose  of  creating  a  persistent  gaming  envi¬ 
ronment  for  SE  is  to  generate  a  sandbox  for  testing  out  new  tactics  in 
conjunction  with  science  and  technology  (S&T)  simultaneously.  Dr. 
Peter  Singer,  director  of  the  Brookings  Institution  21st  Century  Defense 
Initiative  (Unmanned  Systems,  2010),  observes  that  "knowing  that  hav¬ 
ing  the  right  doctrine  can  be  the  difference  between  winning  and  losing 
wars,  between  committing  America  to  the  21st  century  version  of  the 
Maginot  Line  vs.  the  Blitzkrieg.”  SE/2025  has  the  goal  of  generating 
21st  century  blitzkrieg  by  directly  allowing  soldiers  to  experiment  with 
doctrine  directly.  Soldiers  can  then  feed  experiential  insights  and  mea¬ 
surable  data  back  to  engineers  and  decision  makers.  Conversely,  the  art 
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of  the  possible  for  cost,  timing,  and  technology  can  be  provided  back  to 
the  soldiers.  The  speed  of  feedback  produced  in  a  gaming  environment 
suggests  the  potential  for  engineers,  program  offices,  and  soldiers  to 
codevelop  systems.  Gaming  environments  might  even  allow  an  assess¬ 
ment  of  the  battlefield  value  of  S&T  investments  prior  to  committing 
research  dollars  to  actually  develop  the  technology.  The  final  benefit  of 
using  synthetic  environments  is  that  soldiers  will  more  readily  adopt  new 
equipment  if  they  have  already  used  it  in  a  virtual  environment.  It  is  not 
uncommon  for  new  equipment  to  sit  at  the  FOB  because  soldiers  simply 
are  not  comfortable  and  familiar  with  it. 

To  develop  robust  templates  of  the  most  effective  vehicle  con¬ 
figurations,  many  iterations  of  the  same  scenarios  should  be 
performed  due  to  the  stochastic  nature  of  decisions  made  during 
a  battle  (Weber,  2012).  A  slight  deviation  in  timing  or  difference 
in  course-of-action  could  vary  the  battle  outcome  greatly  so  sto¬ 
chastics  are  important.  Another  critical  element  to  maximize  the 
benefit  of  these  environments  is  to  provide  a  discussion  forum 
for  users  to  exchange  tips  and  tricks,  and  to  learn  by  replaying 
winning  and  losing  scenarios.  Collaboration  among  players  will 
ensure  maximum  leapfrogging  of  ideas— known  as  crowd  accel¬ 
erated  innovation  (Anderson,  2010). 

Figure  3  shows  how  a  persistent  gaming  environment  will  engage 
the  Army  DOTmLPF-P  (Doctrine,  Organization,  Training,  materiel. 
Leadership  and  Education,  Personnel,  Facilities  -Policy)  communities. 
The  environment  should  have  several  specific  features:  First,  it  should 
provide  a  sandbox  where  soldiers  may  build  and  modify  ground  systems 
(and  scenarios)  as  they  see  fit.  Second,  the  physics  fidelity  should  be  mod¬ 
ifiable  to  allow  engineers  to  tailor  the  game  with  applicable  real-world 
physics  as  appropriate.  Third,  it  should  be  template-  or  module-centric 
to  avoid  overwhelming  users  with  too  many  combinations  and  choices  so 
they  can  only  focus  on  relevant  details.  Fourth,  there  must  be  a  discussion 
and  sharing  area  that  allows  replays  and  piggybacking  on  ideas. 

3D  Virtual  World  Acquired  on  Demand 

The  spectrum  of  future  operations  covers  a  variety  of  known  and 
unknown  threats,  and  variable  reaction  timelines.  It  is  now  possible  to 
capture,  in  real-time,  a  battle  scenario  that  may  be  input  into  a  gaming 
environment  or  passed  on  to  engineers  for  the  development  of  mission- 
specific  ground  systems.  Planners  for  the  raid  on  Osama  bin  Laden’s 
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Abbottabad,  Pakistan  Compound  used  satellite  imagery  from  the 
National  Geospatial-Intelligence  Agency  to  create  models  of  the  com¬ 
pound  prior  to  the  actual  attack.  The  models  were  used  to  allow  the  Joint 
Special  Operations  Command  to  create  mission  simulators  for  the  pilots 
who  flew  the  helicopters  to  practice  virtually  ahead  of  time  (Ambinder, 
2011;  Harris,  2011).  This  rapid  construction  of  3D  scenarios  will  continue 
to  evolve  and  blur  the  line  between  simulations  and  reality. 

Either  satellite  imagery  or  air-/ground-collected  imagery  may  be 
used  instantly  to  construct  realistic  scenes.  Depending  on  the  applica¬ 
tion,  various  sensor  modalities  maybe  employed.  DARPA  and  Space  and 
Naval  Warfare  Systems  Command  (SPAWAR)  have  both  made  progress 
on  a  number  of  projects  that  help  to  make  instant  scenarios  available.  The 
DARPA  RealWorld  Project  (Intiflc,  n.d.)  has  a  goal  of  creating  high-def¬ 
inition  scenes  in  under  30  minutes.  SPAWAR's  UrbEM  Project  (Nguyen 
et  al.,  2009)  aims  to  develop,  mature,  and  demonstrate  technologies  that 
will  provide  rich  3D  models  of  complex  urban  environments  from  the 
ground  perspective,  mainly  using  sensors  normally  found  on  unmanned 
ground  vehicles.  UrbEM  has  investigated  the  following  technologies 


FIGURE  3.  CONCEPT  OF  THE  PERSISTENT  GAMING 
ENVIRONMENT 
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that  may  be  used  to  develop  scenarios:  structure-from-motion,  multi¬ 
view  stereo,  laser  scanning  fused  with  color  image  data,  spatial  phase 
video,  and  registration  software/algorithms.  An  example  of  an  UrbEM 
experiment  is  shown  in  Figure  4  where  multiple  views  are  automatically 
combined  to  create  a  3D  model.  Similar  effects  may  be  achieved  where 
video  frames  are  continuously  acquired.  The  Microsoft  Photosynth 
project  (Photosynth,  n.d.)  demonstrated  the  ability  to  create  3D  geometry 
from  a  collection  of  online  pictures,  which  is  stunning  considering  it 
requires  a  computer  to  combine  multiple  views,  lens,  lighting,  and  even 
the  inclusion  of  people  in  photographs.  With  Photosynth,  it  is  possible 
simply  to  use  Web-based  photo  repositories  such  as  Google  Images  or 
Flickr  to  create  3D  models  of  objects  autonomously. 

Preengineered  Plug-and-Play  Vehicle  Templates 

A  template  in  the  context  of  future  vehicle  design  is  an  assembly 
of  modules  that  is  a  doctrinal  preference  for  a  successful  outcome. 
Templates  are  key  to  the  rapid  fielding  of  different  solutions  based  on 
terrain,  enemy,  mission,  or  other  considerations.  Imagine  a  case  where 
there  will  be  a  sustained  operation  requiring  the  capture  of  insurgents. 
Users  should  be  able  to  select  preengineered  vehicle  templates  to  try 
out  in  advance  to  see  what  works  best.  Once  they  find  that  a  robot  or 
tank  works  well,  they  can  tailor  the  template  vehicle  to  their  tastes  and 
preferences.  Having  a  generic  starting  template  is  important  in  case  an 
event  occurs  that  requires  an  immediate  response,  allowing  no  time  to 
customize  vehicles  beyond  what  is  captured  already  in  the  template. 
This  is  also  important  for  experimenting  in  the  gaming  environment  so 
players  have  base  vehicles  with  which  to  play  in  the  virtual  environment 
without  starting  from  scratch.  The  development  of  templates  encourages 
innovative  evolution  of  designs  within  the  gaming  environment  by  allow¬ 
ing  easy  modifications.  A  distinct  combat  advantage  is  to  be  gained  from 
tailoring  because  it  will  confound  the  enemy's  ability  to  exploit  a  common 
vulnerability— the  Achilles'  heel  might  always  change. 

Templates,  along  with  modularity,  are  critical  to  avoid  decision 
paralysis  in  the  face  of  too  many  options.  Information  overload  directly 
reduces  the  human  ability  to  make  smart,  creative,  and  successful  deci¬ 
sions  (Begley,  2011).  As  promising  vehicle  configurations  evolve  from 
the  persistent  gaming  environment,  these  can  be  tied  to  classes-of-use 
cases  for  a  vehicle  that  may  be  deployed.  These  configurations  can  then 
be  progressively  tailored  as  more  information  about  a  conflict  becomes 
known  or  the  greater  the  probability  of  a  certain  type  of  event  occurs,  as 
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shown  in  Figure  5.  Individual  commanders  will  be  able  to  customize  the 
base  templates  as  needed  for  specific  missions— be  it  in  the  real  world  or 
virtual  world.  This  evolving  design  methodology  is  supported  by  having 

FIGURE  4.  RECONSTRUCTIONS  OF  3D  MODELS:  STRUCTURE 
AT  CAMP  PENDLETON  MILITARY  OPERATIONS  ON  URBAN 
TERRAIN  COMPOUND 


Note.  Multiple  images  allow  on-the-fly  reconstructions  from  a  series  of  photographs  as 
part  of  the  UrbEM  project.  Adapted  from  Nguyen  et  al.  (2009). 
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discussion  forums  and  replay  capabilities  for  soldiers  to  discuss  what 
options  are  most  desirable  and  to  share  first-person  virtual  operational 
experiences  with  other  stakeholders. 

Detailed  Engineering 

Detailed  engineering  starts  with  Semiautonomous  Virtual  Prototype 
Engineering  (Figure  2).  Virtual  Prototype  denotes  the  fact  that  phys¬ 
ics-based  models  have  already  become  accurate  and  multidisciplinary 
enough  that  they  should  be  considered  digital  (or  virtual)  prototypes. 
Semiautonomous  implies  that  future  modeling  and  simulation  (M&S)  will 
be  more  proactive  than  current  computer  aided  design  (CAD)  since  some 
design  work  can  be  done  collaboratively  with  computers.  Conventional 
M&S  such  as  computational  fluid  dynamics,  finite  element  analysis 
(FE A),  and  other  computer  aided  engineering  methods  are  traditionally 
reactive  and  simply  provide  an  engineer  with  an  assessment  of  perfor¬ 
mance;  the  engineer  must  manually  fix  the  design  and  rerun  the  model. 
For  example,  consider  the  design  of  some  structural  part:  Right  now,  an 
engineer  creates  a  design  in  CAD,  runs  an  FEA  analysis  and,  based  on 
the  results,  repetitively  tweaks  the  design  until  the  part  functions  as 
intended.  In  the  future,  the  engineer  will  merely  describe  the  use-case 
of  the  part  (and  constraints)  to  the  computer  and,  using  M&S,  the  com¬ 
puter  will  autonomously  optimize  the  part.  In  1982,  Gunn  estimated  that 
“only  20%  of  the  parts  initially  thought  to  require  new  designs  actually 
need  them;  40%  could  be  built  from  an  existing  design;  and  40%  could  be 
created  by  modifying  an  existing  design”  (Gunn,  1982).  For  this  reason,  a 
number  of  universities  are  already  working  on  autonomous  part  search 
methodology  (Iyer,  Jayanti,  Lou,  Kalyanaraman,  &  Ramani,  2005). 

The  future  SE  Detailed  Engineering  process  will  be  based  on  per¬ 
vasive  prototyping.  IDEO,  designers  of  Apple's  first  mouse,  the  Gripper 
toothbrush  for  Oral-B,  and  the  Palm  V  point  out  that  “if  a  picture  is  worth 
a  thousand  words,  a  good  prototype  is  worth  a  thousand  pictures”  (Kelley 
&  Littman,  2001).  Prototyping  can  be  virtual  (all  within  a  computer), 
physical,  or  a  combination  of  the  two  such  as  hardware-in-the-loop 
simulations.  Decisions  made  during  detailed  design  must  be  captured 
with  ubiquitous  knowledge  management.  Knowledge  is  expensive  to 
generate  and  ignorance  is  even  more  expensive. 

Physical  prototyping  supplements  pure  M&S  (virtual  prototyping) 
by  validating  assumptions  and  identifying  unknown  interactions.  In 
particular,  subsystem-level  prototypes  can  be  combined  with  modeling 
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to  enable  hardware-in-the-loop  simulations  and  man-in-the-loop  simu¬ 
lations.  The  act  of  building  something  in  itself  is  incredibly  informative. 
Systems  integration  laboratories  (SILs)  also  fall  into  the  category  and 
test  the  integrated  function  of  multiple  components.  SILs  are  critical 
because  SE  fails  most  frequently  at  the  interfaces.  Examples  of  a  physical 
simulation  are  shown  in  Figures  6  and  7. 

Layered  Manufacturing,  Repair,  and  Logistics 

The  future  force  might  be  substantially  redefined  by  new  options  pre¬ 
sented  via  rapid  manufacturing,  and  particularly  additive  manufacturing. 
Per  Wikipedia  (Rapid  Manufacturing,  n.d.),  “Rapid  manufacturing  is  a 

FIGURE  5.  TEMPLATES  OF  MODULAR  VEHICLES  AND  PLAN  FOR 
MANUFACTURING  &  LOGISTICS 


Note.  Such  templates  and  plans  will  become  inseparably  coupled  in  the  future. 
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technique  for  manufacturing  solid  objects  by  the  sequential  delivery  of 
energy  and/or  material  to  specified  points  in  space  to  produce  that  part  ” 
3D  printing  reduces  the  number  of  separate  machines  necessary  to  create  a 
part  by  transforming  powdered  or  liquid  raw  materials  layer  by  layer  into  a 
final  piece.  Additionally,  additive  manufacturing  allows  the  elimination  of 
welding,  brackets,  and  fianges  when  the  piece  can  be  produced  as  a  whole. 
Conventional  machining  processes  remove  material,  which  creates  waste, 
where  additive  manufacturing  only  places  material  where  needed.  Finally, 
additive  manufacturing  may  also  be  used  to  make  repairs.  General  Electric 
has  demonstrated  an  ability  to  repair  worn  parts  by  using  a  precision  spray 
technique  to  add  material  to  an  existing  part  (General  Electric,  2013). 

A  new  ability  to  produce  parts  locally  may  substantially  change  pro¬ 
curement  and  repair  logistics.  Future  logistics  (notionally  illustrated  in 
Figure  8)  must  optimize  the  movement  of  materials  and  manufacturing 
equipment  to  provide  maximum  fiexibility  and  minimal  cost.  Items  that 
require  large  amounts  of  energy,  materials,  and  specialized  environments 
will  likely  be  produced  stateside.  Some  items  may  be  manufactured  at 
the  FOB  using  technologies  such  as  3D  printing.  The  Navy  explored 
the  notion  of  ships  becoming  fioating  factories  in  a  Proceedings  Article 
(Cheney-Peters  &  Hippie,  2013),  possibly  even  harvesting  resources  from 


FIGURE  6.  US  ARMY  TANK  AUTOMOTIVE  RESEARCH, 
DEVELOPMENT  AND  ENGINEERING  CENTER  (TARDEC)’S  RIDE 
MOTION  SIMULATOR  (RMS) 


Note.  TARDEC’s  RMS  is  an  exannple  of  a  nnan-in-the-loop  physical  sinnulation. 
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the  surrounding  seas  or  ashore.  Due  to  the  intrinsic  complexity  of  custom¬ 
ized  platforms,  it  will  be  critical  to  use  information  technology  to  form  an 
effective  manufacturing  and  logistics  strategy. 

Army  Captain  Elsmo  provides  a  simplistic  storyline  example 
(Elsmo,  1999): 

In  reality,  a  ground  system  will  probably  have  multiple  com¬ 
ponents  coming  from  a  variety  of  locations.  Assemblies  and 
subassemblies  may  be  created  anywhere  in  the  logistics  and 
manufacturing  chain.  This  gives  a  very  new  meaning  to  what  the 
life  cycle  of  a  product  and  its  constituent  modules  may  become. 

The  layered  manufacturing/logistics  process  is  tied  directly  to  the 
gaming  environment  and  detailed  engineering  process.  Figure  5  shows 
how  modules  and  developed  templates  for  ground  systems  evolve  in 
lock  step  with  manufacturing  and  logistics.  As  more  information  devel¬ 
ops  about  the  potential  materiel  need,  more  definition  of  the  design  is 
provided.  Once  a  system  has  been  fielded,  modules  allow  a  vehicle  to  be 
adapted  by  changing  out  these  modules.  Examples  include  kittable  armor, 
swapping  out  radios,  upgrading  sensor  packs,  or  retuning  engine  control 
modules.  Further,  the  vehicle  itself  will  be  smart.  An  example  of  a  smart 
vehicle  is  one  that  senses  a  cargo  load  and  then  automatically  reprograms 
its  stability  control  and  antilock  braking  to  accommodate  the  load. 


FIGURE  7.  US  ARMY  TARDECS  N-POST  SHAKER 


Note.  TARDEC’s  N-post  shaker  is  a  hardware-in-the-loop  simulation. 
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to  provide  maximum  flexibility  and  minimal  cost. 
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The  notion  of  local  manufacturing  is  not  entirely  new  to  the 
Army.  The  U.S.  Army  Tank  Automotive  Research,  Development  and 
Engineering  Center  (TARDEC)  had  fielded  a  mobile  parts  hospital  in  the 
past,  which  was  the  automotive  equivalent  to  the  mobile  army  surgical 
hospital  unit,  providing  treatment  to  a  vehicle  so  its  crew  is  protected  and 
could  finish  the  mission  (Williams,  2004).  The  Rapid  Equipping  Force 
began  fielding  expeditionary  lab  mobile  units  in  2013,  which  include 
3D  printers,  computer-assisted  milling  machines,  and  laser,  plasma, 
and  water  cutters,  along  with  common  tools  like  saws  and  welding  gear 
(Hill,  2013).  The  industry  is  fast  approaching  a  point  where  even  static 
structures  such  as  buildings  maybe  3D  printed  (University  of  Southern 
California,  n.d.).  Logistics  must  also  modernize  to  take  advantage  of 
these  new  production  technologies.  Boeing  has  already  used  3D  printing 
to  make  more  than  22,000  parts  used  on  civilian  and  military  aircraft 
fiying  today  (The  Future  of  Military  Logistics,  2012). 

Due  to  changes  in  manufacturing  and  logistics,  the  defense  industry 
could  start  to  shift  away  from  the  historical  big  contract  methodology 
where  large  defense  contractors  are  awarded  a  contract  to  develop  an  entire 
vehicle  based  on  requirements  documents  that  may  exceed  300+  pages. 
In  the  world  of  commercial  automotive,  the  lines  have  already  started  to 
blur  as  to  what  the  brand  name  of  a  vehicle  means.  Engines  come  from  one 
manufacturer,  bodies  another,  and  electronics  another.  Looking  further 
into  the  future,  the  manufacture  of  a  future  ground  vehicle  may  become  a 
very  layered  manufacturing  and  logistics  process.  The  role  for  contractors 
in  such  a  future  may  be  to  develop  modules  and  subsystems  that  plug-and- 
play  with  vehicles.  Additionally,  contractors  might  supply  manufacturing 
equipment  and  maintain  the  logistics  base  that  will  enable  mass  custom¬ 
ization.  Such  a  shift  would  have  an  impact  on  the  planning  and  budgeting 
process,  which  is  focused  on  platforms  in  contrast  to  modules. 

Conclusions 

The  complex  nature  of  future  global  conditions  requires  ground  vehicles 
that  are  adaptable,  fiexible,  smart,  and  rapidly  deployable.  The  very  nature 
of  this  type  of  vehicle  requires  an  agile  SE  process  that  anticipates  many 
scenarios  in  advance.  Using  persistent  synthetic  gaming  environments 
may  help  develop  vehicle  templates  that  consider  tactics  and  technology 
concurrently.  Templates  will  provide  the  most  robust  mission  (and  cost) 
effectiveness  while  still  allowing  for  tailoring.  Rapid  manufacturing  and 
nonstatic  mission  requirements  are  quickly  making  one-size-fits-all  military 
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ground  vehicles  an  obsolete  concept.  Logistics  may  be  transformed  into  a 
deeply  interlinked  manufacturing/repair/logistics  process  with  localized 
production  and  assembly  of  many  parts  or  modules.  Readers  should  consider 
whether  the  next  great  technology  breakthrough  for  the  Army  might  be  an 
agile  systems  engineering  process  that  is  infused  with  crowdsourced  soldier 
input,  concise  communication  of  information,  and  proactive  M&S  tools. 
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Review: 

University  of  Kentucky  Professor  Robert  M.  Farley  has  written  a  very 
controversial  account  of  why  he  believes  the  assets  of  the  U.S.  Air  Force 
should  be  broken  down  and  dispersed  among  the  other  military  services, 
namely  the  army  and  navy,  removing  the  need  for  an  independent  air  force. 
Professor  Farley  includes  a  proposition  for  a  new  structure  of  reorganiza¬ 
tion  for  the  future  of  the  nation's  airpower,  which  would  largely  change  the 
focus  of  the  military  services.  This  review  provides  an  objective  analysis 
of  the  book.  Grounded:  The  Case  for  Abolishing  the  United  States  Air  Force. 

On  one  hand,  the  author  does  a  good  job  of  presenting  his  case  using 
several  instances  in  the  history  of  the  U.S.  Air  Force  where  it  was  not 
able  to  fulfill  part  of  the  initial  intent  of  its  independence,  which  was 
the  ability  to  win  wars  from  the  air  with  very  little  loss  of  life  and  with 
better  cost  efficiency.  Professor  Farley  additionally  does  an  outstanding 
job  of  enlightening  readers  on  how  often  and  how  many  new  aircraft 
are  designed  to  do  a  specific  job,  then  are  deemed  obsolete  due  to  ever- 
evolving  technology  or  are  grounded  for  technical  issues.  Along  with  a 
few  characteristics  of  the  U.S.  Air  Force  that  create  political  and  mili¬ 
tary  problems  for  the  United  States,  he  effectively  lists  four  proposed 
principles  of  reorganization  for  the  removal  of  the  U.S.  Air  Force's  inde¬ 
pendence.  Also  to  the  author's  credit,  he  acknowledges  the  importance 
of  military  aviators  and  the  courage  of  members  of  the  U.S.  Air  Force, 
reiterating  that  Grounded  “should  be  understood  as  part  of  the  opening 
gambit  for  a  restructuring  of  U.S.  military  institutions." 

On  the  other  hand,  perhaps  not  all  bases  were  covered  in  the  consider¬ 
ation  of  the  revamping  of  the  military  services,  which  proposes  to  remove 
the  independence  of  just  one  of  the  Services,  but  would  create  great  debate 
within  the  acquisition  community.  The  question  posed  by  Professor  Farley 
that  seemed  to  jump  out  as  a  basic  concern  is:  “Does  giving  an  air  force 
independence  solve  more  problems  than  it  creates?"  However,  because  our 
independent  air  force  has  now  existed  since  1947,  should  not  the  question 
be:  “Does  removing  the  independence  of  an  air  force  solve  more  problems 
than  it  creates?"  Professor  Farley  does  not  discuss  exactly  by  what  means 
the  nation  will  save  money  should  the  U.S.  Air  Force  be  abolished,  although 
his  plan  addresses  cost  efficiency.  Following  his  plan  for  reorganization,  and 
taking  into  account  his  four  principles,  it  seems  that  in  dividing  all  of  the 
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U.S.  Air  Force's  assets— including  aircraft,  weapons,  personnel,  bases,  and 
missions,  all  mentioned  by  Professor  Farley— the  Department  of  Defense, 
the  nation,  and  the  military  services  would  lose  not  just  money,  but  decades 
of  knowledge  and  expertise  possessed  by  current  members  of  the  U.S.  Air 
Force.  Understandably,  many  airmen  would  either  not  want  to  join  one  of 
the  other  Services  or  would  be  unable  to  retain  their  prior  specialties  in 
the  army  or  navy.  He  does  not  discuss  the  issues  of  training  personnel  or 
the  cost  and  availability  of  supplies  and  uniforms.  There  would  likely  need 
to  be  a  shift  of  responsibilities  for  existing  units  in  the  army  and  navy  to 
provide  ample  air  defense  during  this  restructuring— also  not  discussed. 
Another  important  consideration  would  be  the  loss  of  a  seat  for  the  U.S. 
Air  Force  on  the  Joint  Chiefs  of  Staff,  in  which  the  Service  chiefs  ensure 
the  personnel  readiness,  policy,  planning,  and  training  of  their  respective 
military  services.  However,  once  again,  readers  should  consider  the  book 
an  “opening  gambit”  for  restructuring,  not  the  full  plan. 

After  several  years  of  perfecting  his  argument.  Professor  Farley's 
dispersal  of  U.S.  Air  Force  assets  seems  well  laid  out  between  the  army 
and  navy  for  a  new  kind  of  air  domination.  His  argument  is  supported  with 
detailed  evidence.  Defense  acquisition  professionals  can  benefit  from 
Professor  Farley's  discussion  of  the  Clausewitz  approach  and  his  compari¬ 
son  of  the  existence  of  the  U.S.  Air  Force  to  that  of  the  Royal  Air  Force,  the 
aerial  warfare  branch  of  the  British  Armed  Forces,  which  is  also  the  oldest 
independent  air  force  in  the  world;  and  the  Luftwaffe,  the  aerial  warfare 
branch  of  the  German  Wehrmacht  during  World  War  II.  However,  his  work 
fails  to  recognize  the  limited  gains  and  insurmountable  losses  that  would 
result  from  the  removal  of  the  U.S.  Air  Force's  independence,  and  thus  does 
not  answer  the  question  of  whether  giving  the  U.S.  Air  Force  independence, 
or  taking  it,  would  solve  more  problems  than  it  creates.  It  will  be  interest¬ 
ing  to  see  how  the  rest  of  this  long-running  argument  plays  out  in  regard  to 
defense  acquisition. 


Aleisha  R.  Jenkins-Bey  is  the  assistant  editor  of  the  Defense 
Acquisition  Research  Journal.  Prior  to  her  current  position,  she  was  the 
lead  editor  in  Military  OneSource's  Arlington  office.  She  has  served  as  a 
commissioned  officer  in  the  Air  Defense  Artillery  branch  of  the  U.S.  Army. 
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University  (DAU).  All  submissions  receive  a  blind  review  to  ensure 
impartial  evaluation. 


IN  GENERAL 

We  welcome  submissions  from  anyone  involved  in  the  defense  acqui¬ 
sition  process.  Defense  acquisition  is  defined  as  the  conceptualization, 
initiation,  design,  development,  testing,  contracting,  production,  deploy¬ 
ment,  logistics  support,  modification,  and  disposal  of  weapons  and  other 
systems,  supplies,  or  services  needed  for  a  nation's  defense  and  security, 
or  intended  for  use  to  support  military  missions. 

Research  involves  the  creation  of  new  knowledge.  This  generally 
requires  using  material  from  primary  sources,  including  program  docu¬ 
ments,  policy  papers,  memoranda,  surveys,  interviews,  etc.  Articles  are 
characterized  by  a  systematic  inquiry  into  a  subject  to  discover/revise 
facts  or  theories  with  the  possibility  of  infiuencing  the  development  of 
acquisition  policy  and/or  process. 

We  encourage  prospective  writers  to  coauthor,  adding  depth  to 
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been  previously  published  or  has  expertise  in  the  manuscript's  sub¬ 
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•  discuss  suggestions  for  future  research  (if  applicable). 
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well  as  Web  publication.  Articles  between  4,500  and  10,000  words  will 
be  considered  for  Web-only  publication,  with  an  abstract  included  in  the 
print  version  of  the  Defense  ARJ.  In  no  case  should  article  submissions 
exceed  10,000  words. 

Audience  and  Writing  Styie 

The  readers  of  the  Defense  ARJ  are  primarily  practitioners  within 
the  defense  acquisition  community.  Authors  should  therefore  strive  to 
demonstrate,  clearly  and  concisely,  how  their  work  affects  this  commu¬ 
nity.  At  the  same  time,  do  not  take  an  overly  scholarly  approach  in  either 
content  or  language. 

Format 

Please  submit  your  manuscript  with  references  in  APA  format 
(author-date-page  number  form  of  citation)  as  outlined  in  the  Publication 
Manual  of  the  American  Psychological  Association  [6  th  Edition]).  For  all 
other  style  questions,  please  refer  to  the  Chicago  Manual  of  Style  (16th 
Edition). 

Contributors  are  encouraged  to  seek  the  advice  of  a  reference  librar¬ 
ian  in  completing  citation  of  government  documents  because  standard 
formulas  of  citations  may  provide  incomplete  information  in  reference 
to  government  works.  Helpful  guidance  is  also  available  in  The  Complete 
Guide  to  Citing  Government  Documents  (Revised  Edition):  A  ManwaZ/or 
Writers  and  Librarians  (Garner  &  Smith,  1993),  Bethesda,  Maryland: 
Congressional  Information  Service. 

Pages  should  be  double-spaced  and  organized  in  the  following 
order:  title  page  (titles,  12  words  or  less),  abstract  (150  words  or  less  to 
conform  with  formatting  and  layout  requirements  of  the  publication), 
two-line  summary,  list  of  keywords  (five  words  or  less),  body  of  the  paper, 
reference  list  (only  include  works  cited  in  the  paper),  author's  note  or 
acknowledgments  (if  applicable),  and  figures  or  tables  (if  any). 

Figures  or  tables  should  not  be  inserted  or  embedded  into  the  text, 
but  segregated  (one  to  a  page)  at  the  end  of  the  text.  When  material  is 
submitted  electronically,  each  figure  or  table  should  be  saved  to  a  sepa¬ 
rate,  exportable  file  (i.e.,  a  readable  EPS  file).  For  additional  information 
on  the  preparation  of  figures  or  tables,  refer  to  the  Scientific  Illustration 
Committee,  1988,  Illustrating  Science:  Standards  for  Publication, 
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Copyright 
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Web-only  publications  will  be  held  to  the  same  high  standards  and 
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